Nov 03 00:02:39.872883 ip-10-0-136-97 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 03 00:02:39.872893 ip-10-0-136-97 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 03 00:02:39.872900 ip-10-0-136-97 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 03 00:02:39.873114 ip-10-0-136-97 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 03 00:02:49.911465 ip-10-0-136-97 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 03 00:02:49.911481 ip-10-0-136-97 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 0da4d9095ff04ea5b9422a8140f2c6c7 --
Nov 03 00:04:48.849365 ip-10-0-136-97 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:04:49.243017 ip-10-0-136-97 kubenswrapper[2575]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:49.243017 ip-10-0-136-97 kubenswrapper[2575]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:04:49.243017 ip-10-0-136-97 kubenswrapper[2575]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:49.243017 ip-10-0-136-97 kubenswrapper[2575]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:04:49.249816 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249685    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.249816 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249811    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.249816 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249815    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.249816 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249818    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.249816 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249821    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.249816 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249824    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.249816 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249827    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249829    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249832    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249835    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249837    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249840    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249843    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249845    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249849    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249851    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249854    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249857    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249859    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249862    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249864    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249867    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249869    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249872    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249874    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249877    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.250069 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249879    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249882    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249885    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249887    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249890    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249892    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249895    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249898    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249901    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249903    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249906    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249909    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249912    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249915    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249918    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249921    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249923    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249926    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249928    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.250528 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249931    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249934    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249936    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249939    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249941    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249943    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249946    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249949    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249952    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249954    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249957    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249959    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249962    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249966    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249973    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249976    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249979    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249982    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249985    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.250968 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249987    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249990    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249993    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249995    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.249998    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250001    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250003    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250006    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250009    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250012    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250014    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250016    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250019    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250022    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250024    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250027    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250029    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250032    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250035    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250046    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.251421 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250050    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250070    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250075    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250077    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250080    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250083    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250086    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250088    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250091    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250094    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250096    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250100    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250103    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250105    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250108    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250110    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250113    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250115    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250118    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.251925 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250120    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250123    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250127    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250965    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250972    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250975    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250978    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250982    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250985    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250987    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250990    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250992    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250995    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.250999    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251002    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251005    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251007    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251010    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251013    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251015    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.252398 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251019    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251021    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251024    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251026    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251029    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251034    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251038    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251041    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251043    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251046    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251048    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251051    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251072    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251076    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251079    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251082    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251084    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251087    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251089    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.252921 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251092    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251095    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251097    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251100    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251102    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251105    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251107    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251110    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251113    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251115    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251118    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251120    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251123    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251126    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251129    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251132    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251135    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251138    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251140    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251143    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.253411 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251146    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251148    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251151    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251153    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251156    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251158    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251160    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251163    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251166    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251169    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251171    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251174    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251176    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251179    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251181    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251184    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251186    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251189    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251191    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.253884 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251194    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251197    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251199    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251202    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251204    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251207    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251209    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251212    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251215    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251217    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251220    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251223    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251225    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251228    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251230    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251232    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251236    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251240    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251243    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.254339 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251246    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251248    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251250    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251254    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251256    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251259    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251261    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251264    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251266    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251268    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251271    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251274    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251773    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251779    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251782    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251785    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251788    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251790    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251793    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.254783 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251796    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251798    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251801    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251803    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251809    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251811    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251814    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251816    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251819    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251821    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251824    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251826    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251829    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251831    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251834    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251836    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251839    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251841    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251844    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.255247 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251846    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251849    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251852    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251854    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251857    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251859    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251862    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251864    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251867    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251870    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251872    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251881    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251885    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251889    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251897    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251900    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251903    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251907    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251910    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.255697 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251912    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251915    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251917    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251920    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251922    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251925    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251928    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251930    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251933    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251936    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251939    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251942    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251944    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251946    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251950    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251953    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251955    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251958    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251960    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.256164 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251963    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251965    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251969    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251973    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251976    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251979    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251981    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251984    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251987    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251989    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251991    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251995    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.251998    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252000    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252002    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252005    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252007    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252011    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252013    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.256656 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252016    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252018    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252021    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252023    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252025    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252028    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252030    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252033    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252035    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252037    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252040    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252042    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252045    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252047    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252050    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252052    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252074    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252077    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252080    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252083    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.257119 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252085    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252088    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252090    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252195    2575 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252205    2575 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252212    2575 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252217    2575 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252221    2575 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252224    2575 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252229    2575 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252233    2575 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252236    2575 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252239    2575 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252243    2575 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252246    2575 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252249    2575 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252252    2575 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252255    2575 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252258    2575 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252260    2575 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252263    2575 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252267    2575 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252270    2575 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252273    2575 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:04:49.257588 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252277    2575 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252280    2575 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252284    2575 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252287    2575 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252290    2575 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252293    2575 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252296    2575 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252299    2575 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252302    2575 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252305    2575 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252308    2575 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252312    2575 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252317    2575 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252320    2575 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252322    2575 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252325    2575 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252328    2575 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252334    2575 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252337    2575 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252340    2575 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252342    2575 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252345    2575 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252350    2575 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252352    2575 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252357    2575 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:04:49.258191 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252360    2575 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252363    2575 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252366    2575 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252369    2575 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252371    2575 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252374    2575 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252377    2575 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252380    2575 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252386    2575 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252389    2575 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252392    2575 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252395    2575 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252398    2575 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252402    2575 flags.go:64] FLAG: --help="false"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252405    2575 flags.go:64] FLAG: --hostname-override="ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252408    2575 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252412    2575 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252415    2575 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252418    2575 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252423    2575 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252426    2575 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252429    2575 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252432    2575 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:04:49.258790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252435    2575 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252438    2575 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252441    2575 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252444    2575 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252447    2575 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252450    2575 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252453    2575 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252455    2575 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252460    2575 flags.go:64] FLAG: --lock-file=""
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252463    2575 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252465    2575 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252468    2575 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252474    2575 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252477    2575 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252480    2575 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252483    2575 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252486    2575 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252490    2575 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252492    2575 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252495    2575 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252499    2575 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252503    2575 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252507    2575 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252510    2575 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252513    2575 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:04:49.259347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252516    2575 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252519    2575 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252522    2575 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252526    2575 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252529    2575 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252536    2575 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252540    2575 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252543    2575 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252546    2575 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252548    2575 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252551    2575 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252554    2575 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252557    2575 flags.go:64] FLAG: --port="10250"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252560    2575 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252563    2575 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c6b3e2241db61f1b"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252568    2575 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252571    2575 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252574    2575 flags.go:64] FLAG: --register-node="true"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252577    2575 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252580    2575 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252583    2575 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252586    2575 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252589    2575 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252592    2575 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252595    2575 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252598    2575 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:04:49.259958 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252601    2575 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252603    2575 flags.go:64] FLAG: --runonce="false"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252606    2575 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252609    2575 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252612    2575 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252615    2575 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252618    2575 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252621    2575 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252625    2575 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252629    2575 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252632    2575 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252635    2575 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252638    2575 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252641    2575 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252644    2575 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252647    2575 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252649    2575 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252654    2575 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252657    2575 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252660    2575 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252664    2575 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252668    2575 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252671    2575 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252673    2575 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252676    2575 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:04:49.260583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252679    2575 flags.go:64] FLAG: --v="2"
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252683    2575 flags.go:64] FLAG: --version="false"
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252687    2575 flags.go:64] FLAG: --vmodule=""
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252691    2575 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.252694    2575 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252831    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252834    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252837    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252840    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252842    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252845    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252847    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252850    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252852    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252855    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252858    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252862    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252864    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252868    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252872    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.261274 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252875    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252877    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252880    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252882    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252885    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252887    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252890    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252892    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252896    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252899    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252902    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252904    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252907    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252910    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252912    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252915    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252917    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252920    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252922    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252925    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.261744 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252928    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252931    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252933    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252936    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252938    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252941    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252943    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252946    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252950    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252953    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252956    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252958    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252961    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252963    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252965    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252968    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252971    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252973    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252975    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.262241 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252978    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252981    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252984    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252986    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252989    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252991    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252994    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252997    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.252999    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253001    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253004    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253006    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253009    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253011    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253014    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253017    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253019    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253022    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253024    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.262689 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253026    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253029    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253031    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253035    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253037    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253040    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253042    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253045    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253047    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253049    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253052    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253074    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253078    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253082    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253086    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253088    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253091    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253093    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253096    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.263151 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253099    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253101    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253104    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253106    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253109    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253111    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253114    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253116    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253119    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253121    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253124    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253127    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253129    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253132    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.253159    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253194    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.263725 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253198    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253202    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253205    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253207    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253210    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253213    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253215    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253218    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253221    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253223    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253226    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253228    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253232    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253235    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253238    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253241    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253243    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253246    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253248    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.264137 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253251    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253253    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253256    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253258    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253261    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253263    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253266    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253268    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253270    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253273    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253276    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253278    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253281    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253283    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253287    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253292    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253296    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253298    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253301    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.264577 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253303    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253306    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253308    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253310    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253313    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253315    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253319    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253322    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253324    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253327    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253329    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253332    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253334    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253337    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253339    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253342    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253345    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253348    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253351    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.265035 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253353    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253355    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253358    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253360    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253363    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253365    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253368    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253370    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253372    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253375    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253378    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253382    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253385    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253388    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253390    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253393    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253396    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253398    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253400    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253405    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.265540 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253408    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253410    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253413    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253416    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253419    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253422    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253424    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253427    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253430    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253433    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253436    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253438    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253441    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253443    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253445    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253448    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253450    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253453    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253455    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253458    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.266030 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253460    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253462    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253465    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253468    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253471    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253473    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253476    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.253478    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.260587    2575 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.260605    2575 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.260619    2575 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.260624    2575 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260694    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260700    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260705    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260708    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260710    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260713    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260715    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.266574 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260718    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260720    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260723    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260725    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260728    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260731    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260734    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260737    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260741    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260744    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260747    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260749    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260752    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260754    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260757    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260759    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260762    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260764    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260767    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260769    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.267024 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260772    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260774    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260778    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260781    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260783    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260786    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260788    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260791    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260793    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260795    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260798    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260800    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260803    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260805    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260808    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260810    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260812    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260815    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260818    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260821    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.267554 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260823    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260826    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260829    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260831    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260834    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260836    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260839    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260841    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260843    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260846    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260848    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260851    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260854    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260857    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260860    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260863    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260865    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260868    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260871    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260873    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.268251 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260876    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260878    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260880    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260883    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260885    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260888    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260890    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260893    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260895    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260898    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260900    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260903    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260906    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260908    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260911    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260913    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260916    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260918    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260920    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.268902 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260923    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260925    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260928    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260930    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260932    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260935    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260938    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260940    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260943    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260946    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260948    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260951    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260953    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260956    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260958    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260961    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260963    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260966    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260968    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.269558 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.260970    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.260997    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261161    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261166    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261169    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261172    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261175    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261178    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261180    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261182    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261185    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261187    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261190    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261192    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261195    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261197    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.270020 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261200    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261202    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261205    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261208    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261210    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261212    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261215    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261218    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261220    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261223    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261225    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261228    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261230    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261233    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261235    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261237    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261240    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261242    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261245    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261248    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.270418 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261250    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261253    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261255    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261258    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261260    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261263    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261267    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261271    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261274    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261276    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261279    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261282    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261284    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261287    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261290    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261292    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261295    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261297    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261300    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.270882 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261302    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261305    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261307    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261310    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261312    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261315    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261318    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261321    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261324    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261327    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261329    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261332    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261334    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261337    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261339    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261342    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261344    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261347    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261349    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.271354 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261352    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261354    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261356    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261359    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261362    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261364    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261367    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261369    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261372    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261374    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261377    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261379    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261381    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261384    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261386    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261389    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261392    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261394    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261397    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.271799 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261399    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261401    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261404    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261406    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261409    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261411    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261413    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261418    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261420    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261423    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261425    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261427    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261430    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261432    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261435    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.261457    2575 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:04:49.272347 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261489    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261494    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261496    2575 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261499    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261501    2575 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261504    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261506    2575 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261509    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261511    2575 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261514    2575 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261517    2575 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261521    2575 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261525    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261528    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261531    2575 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261534    2575 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261536    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261539    2575 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261541    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:04:49.272724 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261544    2575 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261546    2575 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261549    2575 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261551    2575 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261555    2575 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261558    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261560    2575 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261563    2575 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261565    2575 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261568    2575 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261570    2575 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261572    2575 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261575    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261577    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261579    2575 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261582    2575 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261584    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261587    2575 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261589    2575 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261592    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:04:49.273211 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261594    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261596    2575 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261599    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261601    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261604    2575 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261606    2575 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261609    2575 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261611    2575 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261614    2575 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261617    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261619    2575 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261621    2575 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261624    2575 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261626    2575 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261628    2575 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261630    2575 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261635    2575 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261637    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261640    2575 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:04:49.273713 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261642    2575 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261644    2575 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261647    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261649    2575 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261652    2575 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261654    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261656    2575 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261659    2575 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261661    2575 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261664    2575 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261666    2575 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261669    2575 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261671    2575 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261673    2575 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261676    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261678    2575 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261680    2575 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261683    2575 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261685    2575 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261687    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:04:49.274191 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261690    2575 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261693    2575 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261695    2575 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261697    2575 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261700    2575 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261702    2575 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261704    2575 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261707    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261709    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261713    2575 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261716    2575 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261719    2575 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261721    2575 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261723    2575 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261733    2575 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261736    2575 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261738    2575 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261741    2575 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261744    2575 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:04:49.274679 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261749    2575 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261754    2575 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261757    2575 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261759    2575 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261762    2575 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261765    2575 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261767    2575 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261770    2575 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:49.261772    2575 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.261826    2575 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.263761    2575 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.264731    2575 server.go:1014] "Starting client certificate rotation"
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.264821    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.264857    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.267020    2575 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:04:49.275134 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.270891    2575 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:04:49.283235 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.283216    2575 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:04:49.286408 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.286393    2575 log.go:25] "Validated CRI v1 image API"
Nov 03 00:04:49.287290 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.287263    2575 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:04:49.289792 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.289759    2575 fs.go:135] Filesystem UUIDs: map[0bfb0e16-cf90-4b9a-895e-e1dd3e36eb27:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0b66fa0-1bd3-4bcb-9fb5-ac2a56b63c80:/dev/nvme0n1p3]
Nov 03 00:04:49.289792 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.289784    2575 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 03 00:04:49.294877 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.294761    2575 manager.go:217] Machine: {Timestamp:2026-11-03 00:04:49.293372994 +0000 UTC m=+0.324745264 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096154 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274b6b8202b92f9a120b00f0fb0c12 SystemUUID:ec274b6b-8202-b92f-9a12-0b00f0fb0c12 BootID:0da4d909-5ff0-4ea5-b942-2a8140f2c6c7 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:8a:73:4d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:8a:73:4d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:de:e9:75:90:61:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:04:49.294877 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.294872    2575 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:04:49.295005 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.294981    2575 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:04:49.297815 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.297788    2575 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:04:49.297973 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.297818    2575 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:04:49.298587 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.298575    2575 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:04:49.298642 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.298593    2575 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:04:49.298642 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.298632    2575 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:04:49.298715 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.298653    2575 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:04:49.299489 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.299477    2575 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:04:49.303298 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.303283    2575 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:04:49.303438 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.303427    2575 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:04:49.304223 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.304213    2575 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:04:49.304278 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.304232    2575 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:04:49.304278 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.304246    2575 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:04:49.304278 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.304260    2575 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:04:49.304278 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.304272    2575 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:04:49.304650 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.304636    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:04:49.307779 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.307755    2575 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:04:49.312847 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.312831    2575 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:04:49.313545 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313533    2575 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:04:49.313583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313563    2575 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:04:49.313734 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313712    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:04:49.313734 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313738    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:04:49.313858 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313744    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:04:49.313858 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313750    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:04:49.313858 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313756    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:04:49.313858 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.313761    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:04:49.314263 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314253    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:04:49.314263 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314263    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:04:49.314339 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314270    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:04:49.314339 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314276    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:04:49.314339 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314285    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:04:49.314339 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314294    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:04:49.314739 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314726    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:04:49.314739 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.314740    2575 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:04:49.318527 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.318512    2575 server.go:1263] "Started kubelet"
Nov 03 00:04:49.318656 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.318605    2575 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:04:49.318719 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.318609    2575 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:04:49.318719 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.318682    2575 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:04:49.319205 ip-10-0-136-97 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:04:49.319764 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.319699    2575 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:04:49.321461 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.321447    2575 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:04:49.324558 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.324535    2575 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 03 00:04:49.324745 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.324724    2575 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:04:49.324745 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.324740    2575 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:04:49.324886 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.324832    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:49.325637 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.325623    2575 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:04:49.325747 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.325625    2575 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:04:49.325814 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.325750    2575 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:04:49.325868 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.325855    2575 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:04:49.325868 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.325865    2575 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:04:49.325975 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.325948    2575 factory.go:55] Registering systemd factory
Nov 03 00:04:49.325975 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.325959    2575 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:04:49.326190 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.326178    2575 factory.go:153] Registering CRI-O factory
Nov 03 00:04:49.326234 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.326192    2575 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:04:49.326263 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.326245    2575 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:04:49.326263 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.326259    2575 factory.go:103] Registering Raw factory
Nov 03 00:04:49.326362 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.326269    2575 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:04:49.327330 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.327317    2575 manager.go:319] Starting recovery of all containers
Nov 03 00:04:49.338294 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.338273    2575 manager.go:324] Recovery completed
Nov 03 00:04:49.339961 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.339921    2575 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Nov 03 00:04:49.343330 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.343312    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:49.345144 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.345115    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:49.345233 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.345157    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:49.345233 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.345172    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:49.345649 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.345631    2575 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:04:49.345649 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.345644    2575 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:04:49.345649 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.345650    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:49.345792 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.345659    2575 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:04:49.346454 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.346429    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:49.346511 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.346458    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:49.346511 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.346472    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:49.347667 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.347653    2575 policy_none.go:50] "Start"
Nov 03 00:04:49.347730 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.347673    2575 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:04:49.347730 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.347686    2575 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:04:49.349613 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.349600    2575 policy_none.go:44] "Start"
Nov 03 00:04:49.358741 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.358717    2575 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-136-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 03 00:04:49.358824 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.358724    2575 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-136-97.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 03 00:04:49.359712 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.358729    2575 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-136-97.ec2.internal.18e4602afb21a735  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-136-97.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-136-97.ec2.internal,},FirstTimestamp:2026-11-03 00:04:49.318479669 +0000 UTC m=+0.349851939,LastTimestamp:2026-11-03 00:04:49.318479669 +0000 UTC m=+0.349851939,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-136-97.ec2.internal,}"
Nov 03 00:04:49.361927 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.361908    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:04:49.371656 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.371510    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 03 00:04:49.371724 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.371527    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-136-97.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 03 00:04:49.371724 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.371535    2575 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 03 00:04:49.372676 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.372660    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w62v7"
Nov 03 00:04:49.377997 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.377981    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w62v7"
Nov 03 00:04:49.382376 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.382351    2575 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:04:49.404433 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.404370    2575 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:04:49.404837 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.404822    2575 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:04:49.404837 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.404837    2575 server.go:87] "Starting device plugin registration server"
Nov 03 00:04:49.405053 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.405041    2575 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:04:49.405152 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.405052    2575 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:04:49.405206 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.405192    2575 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:04:49.405313 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.405279    2575 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:04:49.405313 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.405295    2575 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:04:49.405760 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.405742    2575 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:04:49.405833 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.405789    2575 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:49.469513 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.469485    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:04:49.471261 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.470752    2575 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:04:49.471261 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.470772    2575 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:04:49.471261 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.470789    2575 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:04:49.471261 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.470824    2575 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 03 00:04:49.483233 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.483216    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:49.505857 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.505810    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:49.506680 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.506653    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:49.506680 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.506682    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:49.506791 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.506691    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:49.506791 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.506720    2575 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.523400 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.523382    2575 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.523453 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.523404    2575 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-136-97.ec2.internal\": node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:49.555985 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.555960    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:49.571270 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.571250    2575 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"]
Nov 03 00:04:49.571341 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.571308    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:49.572154 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.572139    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:49.572257 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.572166    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:49.572257 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.572175    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:49.574246 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574232    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:49.574443 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574425    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.574519 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574462    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:49.574899 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574879    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:49.574988 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574913    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:49.574988 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574924    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:49.574988 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574884    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:49.575098 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.574999    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:49.575098 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.575011    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:49.576848 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.576830    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.576887 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.576863    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:49.577515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.577502    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:49.577585 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.577525    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:49.577585 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.577537    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:49.608443 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.608418    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-97.ec2.internal\" not found" node="ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.611182 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.611164    2575 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-136-97.ec2.internal\" not found" node="ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.627007 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.626984    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.627114 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.627010    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.627114 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.627028    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.657078 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.657044    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:49.727167 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.727140    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.727270 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.727171    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.727270 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.727186    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.727270 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.727213    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.727270 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.727245    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.727412 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.727240    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.757401 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.757344    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:49.858008 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.857971    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:49.912176 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.912147    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.914266 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:49.914249    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:49.958109 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:49.958077    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.058723 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.058652    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.159128 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.159105    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.259699 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.259652    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.264517 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.264495    2575 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 03 00:04:50.264700 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.264676    2575 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 03 00:04:50.359825 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.359792    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.374095 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.374067    2575 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:50.382962 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.382922    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:50 +0000 UTC" deadline="2028-08-06 17:47:03.903553398 +0000 UTC"
Nov 03 00:04:50.382962 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.382957    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15425h42m13.52060068s"
Nov 03 00:04:50.388350 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.388330    2575 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:04:50.389675 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.389647    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:04:50.389784 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.389693    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:04:50.389784 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.389706    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:04:50.460527 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.460500    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.474184 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.474162    2575 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:50.475404 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:50.475374    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc90fa10f5e4f787096a878d9a85228.slice/crio-38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c WatchSource:0}: Error finding container 38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c: Status 404 returned error can't find the container with id 38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c
Nov 03 00:04:50.475883 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:50.475867    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ab5ae10a035469fb072566acf5cdc5.slice/crio-ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d WatchSource:0}: Error finding container ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d: Status 404 returned error can't find the container with id ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d
Nov 03 00:04:50.483543 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:50.483522    2575 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:04:50.560891 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.560866    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.661436 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.661380    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.761951 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.761927    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:50.862699 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:50.862669    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:52.556706 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:52.556627    2575 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:04:52.570039 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.569991    2575 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:52.618999 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.618961    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:52.640536 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.640510    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:04:52.641701 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.641675    2575 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:04:52.660952 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.660862    2575 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:04:52.741051 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.741022    2575 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:04:52.899142 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.898858    2575 apiserver.go:52] "Watching apiserver"
Nov 03 00:04:52.919668 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.919020    2575 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 03 00:04:52.936440 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.936415    2575 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:04:52.936806 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.936778    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq","openshift-dns/node-resolver-6lccb","openshift-image-registry/node-ca-wf7kp","openshift-multus/multus-8kjlq","openshift-multus/multus-additional-cni-plugins-bdnqz","openshift-cluster-node-tuning-operator/tuned-k25z9","openshift-network-diagnostics/network-check-target-8l8m5","openshift-multus/network-metrics-daemon-wjrvj","kube-system/global-pull-secret-syncer-xhsst","kube-system/konnectivity-agent-9td9d","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","openshift-network-operator/iptables-alerter-6w7xg","openshift-ovn-kubernetes/ovnkube-node-9dgcv"]
Nov 03 00:04:52.939366 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.939338    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:52.941842 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.941822    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:52.944081 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.944064    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:52.946850 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.946829    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:52.949212 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.949180    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:52.952819 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.952803    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:52.953368 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.953353    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:04:52.955382 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.955328    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqxtt\""
Nov 03 00:04:52.955446 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.955392    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:52.955446 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.955421    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:04:52.955551 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:52.955463    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:04:52.955551 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.955498    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:04:52.955650 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.955567    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:04:52.955650 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.955507    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qr8z8\""
Nov 03 00:04:52.955650 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.955630    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:04:52.957125 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.957106    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzkq\""
Nov 03 00:04:52.957352 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.957336    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:04:52.957440 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.957418    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:04:52.957693 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.957673    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:52.960183 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.960164    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:52.963621 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.963597    2575 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 03 00:04:52.963805 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.963782    2575 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-jl628"
Nov 03 00:04:52.964188 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.964174    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:52.964294 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:52.964235    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:04:52.967037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.967006    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:52.967891 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.967869    2575 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-jl628"
Nov 03 00:04:52.969230 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:52.969208    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:52.969230 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:52.969289    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:04:53.020070 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.020043    2575 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:04:53.028175 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028144    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-hosts-file\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.028310 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028188    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-kubelet\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.028310 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-script-lib\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.028310 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028277    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.028310 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028305    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-agent-certs\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:53.028526 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028339    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-konnectivity-ca\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:53.028526 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028395    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s66w5\" (UniqueName: \"kubernetes.io/projected/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-kube-api-access-s66w5\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.028526 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028442    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd8zx\" (UniqueName: \"kubernetes.io/projected/b24caade-1c7b-44e6-b04e-fbc0c44403fe-kube-api-access-zd8zx\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.028526 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-hostroot\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.028526 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028530    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgdjp\" (UniqueName: \"kubernetes.io/projected/9df7b833-c25f-48d0-8de2-401b05bd6796-kube-api-access-rgdjp\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028556    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-kubelet-config\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028596    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-tuned\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028613    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-multus-daemon-config\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-multus-certs\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028655    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxxbd\" (UniqueName: \"kubernetes.io/projected/3ce18949-1971-4017-ab1c-5faa6d7412b2-kube-api-access-zxxbd\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028681    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-iptables-alerter-script\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028703    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:53.028777 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028783    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-socket-dir-parent\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028840    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-slash\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028895    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.028922    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-dbus\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029009    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029036    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba3da01-3b60-48ec-8d2d-2d90478e334a-host\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029073    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fba3da01-3b60-48ec-8d2d-2d90478e334a-serviceca\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029110    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-tmp-dir\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.029185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029145    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029194    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-log-socket\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029224    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-sys-fs\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029250    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-conf\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029300    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-host\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029326    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029360    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zglcx\" (UniqueName: \"kubernetes.io/projected/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-kube-api-access-zglcx\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-modprobe-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-kubernetes\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029460    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59sh\" (UniqueName: \"kubernetes.io/projected/fba3da01-3b60-48ec-8d2d-2d90478e334a-kube-api-access-l59sh\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-conf-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029533    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-bin\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029575    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysconfig\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.029623 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029624    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j7s6\" (UniqueName: \"kubernetes.io/projected/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-kube-api-access-9j7s6\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029656    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-system-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029713    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-netns\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029751    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-systemd-units\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029819    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-netns\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-config\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029889    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-systemd\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029928    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjr79\" (UniqueName: \"kubernetes.io/projected/b3876106-cc95-464e-9f2c-be6504f26585-kube-api-access-hjr79\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029956    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cnibin\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.029992    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-ovn\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030024    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h6zb\" (UniqueName: \"kubernetes.io/projected/1bf749d7-4d5c-4855-b945-fbf6b4953624-kube-api-access-9h6zb\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030050    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-registration-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030120    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-var-lib-kubelet\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030162    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-system-cni-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-systemd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.030245 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030227    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-etc-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030270    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-device-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030296    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-run\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030318    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-lib-modules\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030371    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-host-slash\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030398    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030422    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030445    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-netd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030471    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030496    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-multus\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030535    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030565    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-cnibin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030628    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-cni-binary-copy\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030667    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-k8s-cni-cncf-io\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030696    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-node-log\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030721    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-socket-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.031002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-sys\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030765    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-bin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030797    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-etc-kubernetes\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030821    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-var-lib-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030850    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-env-overrides\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030874    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030896    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030917    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-tmp\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.031781 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.030953    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-kubelet\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.053649 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.053621    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.053879    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.053933    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.053957    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054023    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054041    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054076    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054134    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054134    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054327    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054439    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054520    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054541    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054543    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054614    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:04:53.054639 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.054632    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:04:53.059503 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.059480    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:04:53.059618 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.059491    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:04:53.059685 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.059673    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xfsv5\""
Nov 03 00:04:53.061586 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.061543    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:04:53.061699 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.061583    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9mnsd\""
Nov 03 00:04:53.061699 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.061652    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nrnbx\""
Nov 03 00:04:53.065448 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.065431    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:04:53.065545 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.065528    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pr6ld\""
Nov 03 00:04:53.065895 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.065874    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jtdbf\""
Nov 03 00:04:53.066113 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.066095    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gt4zj\""
Nov 03 00:04:53.069676 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.069620    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d"}
Nov 03 00:04:53.071145 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.071121    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c"}
Nov 03 00:04:53.132049 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.132049 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-agent-certs\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:53.132305 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132163    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-konnectivity-ca\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:53.132305 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132198    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.132305 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132196    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s66w5\" (UniqueName: \"kubernetes.io/projected/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-kube-api-access-s66w5\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.132305 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132279    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8zx\" (UniqueName: \"kubernetes.io/projected/b24caade-1c7b-44e6-b04e-fbc0c44403fe-kube-api-access-zd8zx\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.132305 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132296    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-hostroot\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132313    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgdjp\" (UniqueName: \"kubernetes.io/projected/9df7b833-c25f-48d0-8de2-401b05bd6796-kube-api-access-rgdjp\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132345    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-kubelet-config\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-tuned\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132372    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-multus-daemon-config\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132399    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-multus-certs\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132418    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxbd\" (UniqueName: \"kubernetes.io/projected/3ce18949-1971-4017-ab1c-5faa6d7412b2-kube-api-access-zxxbd\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-iptables-alerter-script\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132476    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132514    2575 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132525    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-kubelet-config\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.132555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132537    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-multus-certs\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132659    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-socket-dir-parent\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132687    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132944    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-hostroot\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-socket-dir-parent\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.132994    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-slash\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133022    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.133033    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133046    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-dbus\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133070    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133075    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-konnectivity-ca\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:53.133148 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.133119    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs podName:3ce18949-1971-4017-ab1c-5faa6d7412b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:53.633075929 +0000 UTC m=+3.071251295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs") pod "network-metrics-daemon-wjrvj" (UID: "3ce18949-1971-4017-ab1c-5faa6d7412b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133187    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba3da01-3b60-48ec-8d2d-2d90478e334a-host\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.133219    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.133280    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret podName:7ad1f241-5476-4a43-a616-f26eff05c43d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:53.633246346 +0000 UTC m=+3.071421693 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret") pod "global-pull-secret-syncer-xhsst" (UID: "7ad1f241-5476-4a43-a616-f26eff05c43d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133283    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fba3da01-3b60-48ec-8d2d-2d90478e334a-serviceca\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133330    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-tmp-dir\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133358    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133384    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-log-socket\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133408    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-sys-fs\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133431    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-conf\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133452    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-host\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133474    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133485    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-iptables-alerter-script\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133498    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglcx\" (UniqueName: \"kubernetes.io/projected/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-kube-api-access-zglcx\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133522    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-modprobe-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133545    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-kubernetes\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.133606 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133570    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59sh\" (UniqueName: \"kubernetes.io/projected/fba3da01-3b60-48ec-8d2d-2d90478e334a-kube-api-access-l59sh\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133593    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-conf-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133630    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-bin\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133633    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fba3da01-3b60-48ec-8d2d-2d90478e334a-serviceca\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133654    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysconfig\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133668    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba3da01-3b60-48ec-8d2d-2d90478e334a-host\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133680    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j7s6\" (UniqueName: \"kubernetes.io/projected/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-kube-api-access-9j7s6\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133698    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-slash\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-system-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133739    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-log-socket\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133762    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-netns\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133772    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-multus-daemon-config\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-systemd-units\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133813    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-netns\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133836    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-config\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133862    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-systemd\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133887    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjr79\" (UniqueName: \"kubernetes.io/projected/b3876106-cc95-464e-9f2c-be6504f26585-kube-api-access-hjr79\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133908    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-conf\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.134515 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cnibin\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133951    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cnibin\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133958    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-dbus\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133957    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-ovn\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133988    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-ovn\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133997    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-netns\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133991    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6zb\" (UniqueName: \"kubernetes.io/projected/1bf749d7-4d5c-4855-b945-fbf6b4953624-kube-api-access-9h6zb\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134017    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134029    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134053    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-systemd-units\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134053    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-registration-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134086    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-var-lib-kubelet\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134091    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-registration-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134113    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-system-cni-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134139    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-systemd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134152    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-system-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134142    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.135356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134164    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-etc-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134197    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-netns\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134198    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-device-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134203    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-etc-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134247    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-systemd\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.133788    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-sys-fs\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134316    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-run\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134318    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-modprobe-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134335    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-lib-modules\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134356    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-host-slash\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134393    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134461    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-bin\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134519    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-conf-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134561    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-netd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134567    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-device-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134591    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134622    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-multus\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134641    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-run\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134648    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134684    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-cnibin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134697    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-kubernetes\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134708    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-systemd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134714    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-cni-binary-copy\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134740    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-k8s-cni-cncf-io\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134755    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-host\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134767    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-node-log\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134786    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134790    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-socket-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134799    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysconfig\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134823    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-sys\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134840    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-bin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134855    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-etc-kubernetes\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134854    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134871    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-var-lib-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.136854 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134891    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-var-lib-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134895    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-socket-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134899    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-env-overrides\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134920    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-sys\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134839    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-host-slash\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134956    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134961    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-tmp-dir\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134978    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-netd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134982    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135000    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-system-cni-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135004    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-tmp\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135027    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-node-log\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135030    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-kubelet\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135037    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.134960    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-etc-kubernetes\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135055    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-hosts-file\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135084    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-kubelet\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135097    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-cnibin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.137567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-script-lib\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135134    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-bin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135237    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135302    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-multus\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135362    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-k8s-cni-cncf-io\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135407    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-kubelet\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135452    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-cni-binary-copy\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135052    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-var-lib-kubelet\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135508    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135563    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-hosts-file\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135585    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135614    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-kubelet\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.135674    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-lib-modules\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.136138    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-config\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.138170 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.136580    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-env-overrides\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.139467 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.139442    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-tmp\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.139746 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.139723    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.140649 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.140630    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-agent-certs\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:53.140772 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.140752    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-script-lib\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.140928 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.140893    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-tuned\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.152680 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.152155    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:53.152680 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.152182    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:53.152680 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.152200    2575 projected.go:196] Error preparing data for projected volume kube-api-access-8xp29 for pod openshift-network-diagnostics/network-check-target-8l8m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:53.152680 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.152329    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29 podName:04183b41-8425-43cd-a145-9e454673cad1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:53.652305156 +0000 UTC m=+3.090480506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8xp29" (UniqueName: "kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29") pod "network-check-target-8l8m5" (UID: "04183b41-8425-43cd-a145-9e454673cad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:53.152680 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.152398    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgdjp\" (UniqueName: \"kubernetes.io/projected/9df7b833-c25f-48d0-8de2-401b05bd6796-kube-api-access-rgdjp\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.152680 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.152440    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s66w5\" (UniqueName: \"kubernetes.io/projected/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-kube-api-access-s66w5\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.152680 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.152556    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxbd\" (UniqueName: \"kubernetes.io/projected/3ce18949-1971-4017-ab1c-5faa6d7412b2-kube-api-access-zxxbd\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:53.155334 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.155289    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjr79\" (UniqueName: \"kubernetes.io/projected/b3876106-cc95-464e-9f2c-be6504f26585-kube-api-access-hjr79\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.155930 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.155861    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8zx\" (UniqueName: \"kubernetes.io/projected/b24caade-1c7b-44e6-b04e-fbc0c44403fe-kube-api-access-zd8zx\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.155930 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.155872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglcx\" (UniqueName: \"kubernetes.io/projected/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-kube-api-access-zglcx\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.156246 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.156232    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6zb\" (UniqueName: \"kubernetes.io/projected/1bf749d7-4d5c-4855-b945-fbf6b4953624-kube-api-access-9h6zb\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.157156 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.157111    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59sh\" (UniqueName: \"kubernetes.io/projected/fba3da01-3b60-48ec-8d2d-2d90478e334a-kube-api-access-l59sh\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.157512 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.157485    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j7s6\" (UniqueName: \"kubernetes.io/projected/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-kube-api-access-9j7s6\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.256292 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.256168    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:04:53.265058 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.265029    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:04:53.268821 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.268793    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24caade_1c7b_44e6_b04e_fbc0c44403fe.slice/crio-374d16e7d0bae76e67cf56909d957730723eb79a8964e862d7f669b1ca29a429 WatchSource:0}: Error finding container 374d16e7d0bae76e67cf56909d957730723eb79a8964e862d7f669b1ca29a429: Status 404 returned error can't find the container with id 374d16e7d0bae76e67cf56909d957730723eb79a8964e862d7f669b1ca29a429
Nov 03 00:04:53.274620 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.274595    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83c6f1f9_3e2f_4c6e_801f_29735ad355a2.slice/crio-4f29e92e99c391a6c65c933169ce7ac848971d22c52bcfc83318f74fb8117f67 WatchSource:0}: Error finding container 4f29e92e99c391a6c65c933169ce7ac848971d22c52bcfc83318f74fb8117f67: Status 404 returned error can't find the container with id 4f29e92e99c391a6c65c933169ce7ac848971d22c52bcfc83318f74fb8117f67
Nov 03 00:04:53.274717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.274627    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:04:53.280651 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.280514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:04:53.285468 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.285442    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb2a078c_bf02_4c8b_aeec_cbd96df8b1a0.slice/crio-77688afd59189f92aecc7e39fed0e42cff03efa97ea1cdc4a5f9d1eec716bb41 WatchSource:0}: Error finding container 77688afd59189f92aecc7e39fed0e42cff03efa97ea1cdc4a5f9d1eec716bb41: Status 404 returned error can't find the container with id 77688afd59189f92aecc7e39fed0e42cff03efa97ea1cdc4a5f9d1eec716bb41
Nov 03 00:04:53.287646 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.287493    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-8kjlq"
Nov 03 00:04:53.291774 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.291748    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfba3da01_3b60_48ec_8d2d_2d90478e334a.slice/crio-a0f0b7039e12be7087099855e4e23ed82a0d1a8e3e0236eff40ce66c6f4fabba WatchSource:0}: Error finding container a0f0b7039e12be7087099855e4e23ed82a0d1a8e3e0236eff40ce66c6f4fabba: Status 404 returned error can't find the container with id a0f0b7039e12be7087099855e4e23ed82a0d1a8e3e0236eff40ce66c6f4fabba
Nov 03 00:04:53.297985 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.297960    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3876106_cc95_464e_9f2c_be6504f26585.slice/crio-ac7ef5ec05a7c587e3d3839a4f09664c26035bb9b2cf9e4c1e3412a9b0d01648 WatchSource:0}: Error finding container ac7ef5ec05a7c587e3d3839a4f09664c26035bb9b2cf9e4c1e3412a9b0d01648: Status 404 returned error can't find the container with id ac7ef5ec05a7c587e3d3839a4f09664c26035bb9b2cf9e4c1e3412a9b0d01648
Nov 03 00:04:53.316341 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.316303    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:04:53.318729 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.318702    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:04:53.327934 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.327915    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:04:53.329036 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.328689    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21ef2aeb_b54a_45f5_b58a_af731ebdfe11.slice/crio-c45cec789f1f012c5bd82b86ecc1a9c5ecc18a047d585f47513efbd0935a40d3 WatchSource:0}: Error finding container c45cec789f1f012c5bd82b86ecc1a9c5ecc18a047d585f47513efbd0935a40d3: Status 404 returned error can't find the container with id c45cec789f1f012c5bd82b86ecc1a9c5ecc18a047d585f47513efbd0935a40d3
Nov 03 00:04:53.331538 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.331504    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08c0805_8fa0_498f_a60c_9f24b8c6ddd2.slice/crio-44f13221d6eab5752d88f5359f6df20161e29d0c61b4a859303519a6b1ccfb5a WatchSource:0}: Error finding container 44f13221d6eab5752d88f5359f6df20161e29d0c61b4a859303519a6b1ccfb5a: Status 404 returned error can't find the container with id 44f13221d6eab5752d88f5359f6df20161e29d0c61b4a859303519a6b1ccfb5a
Nov 03 00:04:53.334760 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.334672    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:04:53.346605 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:04:53.346579    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bf749d7_4d5c_4855_b945_fbf6b4953624.slice/crio-56b6fb778b1f8ef1b42a565ce08a5e0cc9bf01ac587771f5c41d878f7857d5d7 WatchSource:0}: Error finding container 56b6fb778b1f8ef1b42a565ce08a5e0cc9bf01ac587771f5c41d878f7857d5d7: Status 404 returned error can't find the container with id 56b6fb778b1f8ef1b42a565ce08a5e0cc9bf01ac587771f5c41d878f7857d5d7
Nov 03 00:04:53.638689 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.638667    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:53.638765 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.638736    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:53.638839 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.638820    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:53.638839 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.638834    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:53.638940 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.638889    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs podName:3ce18949-1971-4017-ab1c-5faa6d7412b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.638872523 +0000 UTC m=+4.077047871 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs") pod "network-metrics-daemon-wjrvj" (UID: "3ce18949-1971-4017-ab1c-5faa6d7412b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:53.638940 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.638916    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret podName:7ad1f241-5476-4a43-a616-f26eff05c43d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.638903253 +0000 UTC m=+4.077078601 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret") pod "global-pull-secret-syncer-xhsst" (UID: "7ad1f241-5476-4a43-a616-f26eff05c43d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:53.739362 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.739108    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:53.739475 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.739305    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:53.739475 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.739410    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:53.739475 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.739422    2575 projected.go:196] Error preparing data for projected volume kube-api-access-8xp29 for pod openshift-network-diagnostics/network-check-target-8l8m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:53.739475 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:53.739464    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29 podName:04183b41-8425-43cd-a145-9e454673cad1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:54.739451729 +0000 UTC m=+4.177627075 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8xp29" (UniqueName: "kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29") pod "network-check-target-8l8m5" (UID: "04183b41-8425-43cd-a145-9e454673cad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:53.971434 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.971267    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:52 +0000 UTC" deadline="2028-07-31 14:53:17.086453864 +0000 UTC"
Nov 03 00:04:53.971434 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:53.971328    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15278h48m23.115129246s"
Nov 03 00:04:54.064537 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.064504    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:54.064722 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.064653    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:04:54.065227 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.065205    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:54.065370 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.065324    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:04:54.084985 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.084952    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6lccb" event={"ID":"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0","Type":"ContainerStarted","Data":"77688afd59189f92aecc7e39fed0e42cff03efa97ea1cdc4a5f9d1eec716bb41"}
Nov 03 00:04:54.090241 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.090143    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6w7xg" event={"ID":"83c6f1f9-3e2f-4c6e-801f-29735ad355a2","Type":"ContainerStarted","Data":"4f29e92e99c391a6c65c933169ce7ac848971d22c52bcfc83318f74fb8117f67"}
Nov 03 00:04:54.095921 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.095455    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"be3932b66fe25adccbe122883139881420ac0eb5bbf9813323661d8dc1b04798"}
Nov 03 00:04:54.104482 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.103865    2575 generic.go:356] "Generic (PLEG): container finished" podID="3fc90fa10f5e4f787096a878d9a85228" containerID="d635ceaa75433184b8abf679a600e287e563ba1a6ad46dbfc3a39c249a3ef91a" exitCode=0
Nov 03 00:04:54.104482 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.103938    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerDied","Data":"d635ceaa75433184b8abf679a600e287e563ba1a6ad46dbfc3a39c249a3ef91a"}
Nov 03 00:04:54.123311 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.123276    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"56b6fb778b1f8ef1b42a565ce08a5e0cc9bf01ac587771f5c41d878f7857d5d7"}
Nov 03 00:04:54.139992 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.139962    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9td9d" event={"ID":"21ef2aeb-b54a-45f5-b58a-af731ebdfe11","Type":"ContainerStarted","Data":"c45cec789f1f012c5bd82b86ecc1a9c5ecc18a047d585f47513efbd0935a40d3"}
Nov 03 00:04:54.167425 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.167367    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" podStartSLOduration=2.167348668 podStartE2EDuration="2.167348668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:54.131674526 +0000 UTC m=+3.569849896" watchObservedRunningTime="2026-11-03 00:04:54.167348668 +0000 UTC m=+3.605524038"
Nov 03 00:04:54.170587 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.170554    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kjlq" event={"ID":"b3876106-cc95-464e-9f2c-be6504f26585","Type":"ContainerStarted","Data":"ac7ef5ec05a7c587e3d3839a4f09664c26035bb9b2cf9e4c1e3412a9b0d01648"}
Nov 03 00:04:54.182026 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.181930    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k25z9" event={"ID":"b24caade-1c7b-44e6-b04e-fbc0c44403fe","Type":"ContainerStarted","Data":"374d16e7d0bae76e67cf56909d957730723eb79a8964e862d7f669b1ca29a429"}
Nov 03 00:04:54.195439 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.195363    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq" event={"ID":"9df7b833-c25f-48d0-8de2-401b05bd6796","Type":"ContainerStarted","Data":"0b12439d2683c63cd82aaaab94c427b53e2049e0fa694de58d4b06d285c85921"}
Nov 03 00:04:54.202287 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.202194    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerStarted","Data":"44f13221d6eab5752d88f5359f6df20161e29d0c61b4a859303519a6b1ccfb5a"}
Nov 03 00:04:54.213429 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.213398    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wf7kp" event={"ID":"fba3da01-3b60-48ec-8d2d-2d90478e334a","Type":"ContainerStarted","Data":"a0f0b7039e12be7087099855e4e23ed82a0d1a8e3e0236eff40ce66c6f4fabba"}
Nov 03 00:04:54.648443 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.648406    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:54.648553 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.648473    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:54.648653 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.648633    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:54.648714 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.648703    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret podName:7ad1f241-5476-4a43-a616-f26eff05c43d nodeName:}" failed. No retries permitted until 2026-11-03 00:04:56.648684945 +0000 UTC m=+6.086860295 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret") pod "global-pull-secret-syncer-xhsst" (UID: "7ad1f241-5476-4a43-a616-f26eff05c43d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:54.649129 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.649110    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:54.649211 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.649168    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs podName:3ce18949-1971-4017-ab1c-5faa6d7412b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:56.649152417 +0000 UTC m=+6.087327766 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs") pod "network-metrics-daemon-wjrvj" (UID: "3ce18949-1971-4017-ab1c-5faa6d7412b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:54.749080 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.749047    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:54.749273 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.749228    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:54.749273 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.749248    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:54.749419 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.749277    2575 projected.go:196] Error preparing data for projected volume kube-api-access-8xp29 for pod openshift-network-diagnostics/network-check-target-8l8m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:54.749419 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:54.749333    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29 podName:04183b41-8425-43cd-a145-9e454673cad1 nodeName:}" failed. No retries permitted until 2026-11-03 00:04:56.749315184 +0000 UTC m=+6.187490535 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8xp29" (UniqueName: "kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29") pod "network-check-target-8l8m5" (UID: "04183b41-8425-43cd-a145-9e454673cad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:54.971951 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.971814    2575 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:52 +0000 UTC" deadline="2028-06-27 09:11:31.271509167 +0000 UTC"
Nov 03 00:04:54.971951 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:54.971876    2575 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14457h6m36.299637736s"
Nov 03 00:04:55.069849 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:55.069821    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:55.070023 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:55.069967    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:04:55.238104 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:55.238014    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"acca60ee61ec507f143dbc896b65cbc1bd6c37ead3acc5cce4e145206b1f3032"}
Nov 03 00:04:55.274061 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:55.273897    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" podStartSLOduration=3.273879747 podStartE2EDuration="3.273879747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:55.27290669 +0000 UTC m=+4.711082061" watchObservedRunningTime="2026-11-03 00:04:55.273879747 +0000 UTC m=+4.712055117"
Nov 03 00:04:56.065028 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:56.064507    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:56.065028 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.064676    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:04:56.065028 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:56.064707    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:56.065028 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.064829    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:04:56.244334 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:56.243716    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="4c750a2a345b8d0ac35d83d3537dfc73afa0e6f24b39f85c3212e5ad86bee93f" exitCode=0
Nov 03 00:04:56.244619 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:56.244551    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerDied","Data":"4c750a2a345b8d0ac35d83d3537dfc73afa0e6f24b39f85c3212e5ad86bee93f"}
Nov 03 00:04:56.665862 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:56.665741    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:56.665862 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:56.665811    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:56.666056 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.665953    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:56.666056 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.666021    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret podName:7ad1f241-5476-4a43-a616-f26eff05c43d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.666001796 +0000 UTC m=+10.104177157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret") pod "global-pull-secret-syncer-xhsst" (UID: "7ad1f241-5476-4a43-a616-f26eff05c43d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:04:56.666465 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.666442    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:56.666572 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.666500    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs podName:3ce18949-1971-4017-ab1c-5faa6d7412b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.666486114 +0000 UTC m=+10.104661476 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs") pod "network-metrics-daemon-wjrvj" (UID: "3ce18949-1971-4017-ab1c-5faa6d7412b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:04:56.766185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:56.766146    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:56.766383 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.766353    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:04:56.766383 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.766371    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:04:56.766383 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.766385    2575 projected.go:196] Error preparing data for projected volume kube-api-access-8xp29 for pod openshift-network-diagnostics/network-check-target-8l8m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:56.766540 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:56.766450    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29 podName:04183b41-8425-43cd-a145-9e454673cad1 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:00.766430962 +0000 UTC m=+10.204606330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8xp29" (UniqueName: "kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29") pod "network-check-target-8l8m5" (UID: "04183b41-8425-43cd-a145-9e454673cad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:04:57.065596 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:57.065573    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:57.065990 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:57.065710    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:04:57.248661 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:57.247981    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9td9d" event={"ID":"21ef2aeb-b54a-45f5-b58a-af731ebdfe11","Type":"ContainerStarted","Data":"68a939061553c8eb3be5ca3b9b7b498e74044df83b08b234bb513f10f30472a0"}
Nov 03 00:04:58.064997 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:58.064487    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:04:58.064997 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:58.064620    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:04:58.064997 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:58.064690    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:04:58.064997 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:58.064758    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:04:59.067490 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:59.067405    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:04:59.067900 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:04:59.067557    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:04:59.254270 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:59.254165    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq" event={"ID":"9df7b833-c25f-48d0-8de2-401b05bd6796","Type":"ContainerStarted","Data":"a09bb83d19e1818edb740698f4dbac4cf8b28437236880e617924ddcad3213da"}
Nov 03 00:04:59.259285 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:59.258398    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wf7kp" event={"ID":"fba3da01-3b60-48ec-8d2d-2d90478e334a","Type":"ContainerStarted","Data":"783e6fcd144b73bd0ee7099cfd953dea7fae68f9c807314f6def34f6381868ea"}
Nov 03 00:04:59.297126 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:59.296481    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wf7kp" podStartSLOduration=3.513466426 podStartE2EDuration="8.296462557s" totalImagesPullingTime="4.782996131s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:59.296015703 +0000 UTC m=+8.734191075" watchObservedRunningTime="2026-11-03 00:04:59.296462557 +0000 UTC m=+8.734637928"
Nov 03 00:04:59.297126 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:04:59.296892    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9td9d" podStartSLOduration=4.758250482 podStartE2EDuration="8.296878859s" totalImagesPullingTime="3.538628377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:04:57.285754697 +0000 UTC m=+6.723930067" watchObservedRunningTime="2026-11-03 00:04:59.296878859 +0000 UTC m=+8.735054254"
Nov 03 00:05:00.064952 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:00.064914    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:00.065129 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.065056    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:00.065503 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:00.065467    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:00.065605 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.065568    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:00.644050 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:00.644015    2575 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:05:00.697239 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:00.697203    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:00.697239 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:00.697290    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:00.697551 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.697477    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:05:00.697551 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.697546    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret podName:7ad1f241-5476-4a43-a616-f26eff05c43d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.697524264 +0000 UTC m=+18.135699630 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret") pod "global-pull-secret-syncer-xhsst" (UID: "7ad1f241-5476-4a43-a616-f26eff05c43d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:05:00.697921 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.697905    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:05:00.698012 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.697958    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs podName:3ce18949-1971-4017-ab1c-5faa6d7412b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.697942689 +0000 UTC m=+18.136118054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs") pod "network-metrics-daemon-wjrvj" (UID: "3ce18949-1971-4017-ab1c-5faa6d7412b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:05:00.798275 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:00.797665    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:00.798275 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.797850    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:05:00.798275 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.797868    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:05:00.798275 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.797880    2575 projected.go:196] Error preparing data for projected volume kube-api-access-8xp29 for pod openshift-network-diagnostics/network-check-target-8l8m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:05:00.798275 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:00.797933    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29 podName:04183b41-8425-43cd-a145-9e454673cad1 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:08.797915775 +0000 UTC m=+18.236091151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8xp29" (UniqueName: "kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29") pod "network-check-target-8l8m5" (UID: "04183b41-8425-43cd-a145-9e454673cad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:05:01.005963 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.004507    2575 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:05:00.644035361Z","UUID":"e3b8d3cc-ac51-4420-8f1c-66617c85e423","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:05:01.007808 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.007347    2575 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:05:01.007808 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.007376    2575 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:05:01.065718 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.065684    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:01.065886 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:01.065787    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:01.267320 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.267010    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-8kjlq" event={"ID":"b3876106-cc95-464e-9f2c-be6504f26585","Type":"ContainerStarted","Data":"fa2f3f128954c1548a3670099d0cd935249a9014a1b8dbd81a50c2cf495157b2"}
Nov 03 00:05:01.269837 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.269806    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq" event={"ID":"9df7b833-c25f-48d0-8de2-401b05bd6796","Type":"ContainerStarted","Data":"3f6177234befd3c2c044dbf4bfcd1b5b49a3f72920a7b1d50d7d86d7de2a5f7d"}
Nov 03 00:05:01.272728 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.272143    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="567a85ddaa1fb324ed598e190fbdbf0576523c7f6d6cf6cf0c669b2d0eb0a6c5" exitCode=0
Nov 03 00:05:01.272728 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.272237    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerDied","Data":"567a85ddaa1fb324ed598e190fbdbf0576523c7f6d6cf6cf0c669b2d0eb0a6c5"}
Nov 03 00:05:01.274040 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.274015    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6lccb" event={"ID":"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0","Type":"ContainerStarted","Data":"57e26b704618fd97f8bf9b34aebedb67beece1e96cdff90bb691b083b6cb0069"}
Nov 03 00:05:01.351651 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.351246    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-8kjlq" podStartSLOduration=2.8915850560000003 podStartE2EDuration="10.351232654s" totalImagesPullingTime="7.459647598s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:01.309528782 +0000 UTC m=+10.747704164" watchObservedRunningTime="2026-11-03 00:05:01.351232654 +0000 UTC m=+10.789408024"
Nov 03 00:05:01.385454 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:01.385398    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-6lccb" podStartSLOduration=2.684025395 podStartE2EDuration="10.385377778s" totalImagesPullingTime="7.701352383s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:01.385283081 +0000 UTC m=+10.823458453" watchObservedRunningTime="2026-11-03 00:05:01.385377778 +0000 UTC m=+10.823553149"
Nov 03 00:05:02.064553 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.064525    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:02.064998 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:02.064654    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:02.064998 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.064704    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:02.064998 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:02.064817    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:02.243142 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.243050    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:05:02.243715 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.243690    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:05:02.279247 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.279206    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="727362b073d706a7852af53e1e463a4a9924722ebbd938cf8e13d4b8d79cb027" exitCode=0
Nov 03 00:05:02.279627 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.279604    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerDied","Data":"727362b073d706a7852af53e1e463a4a9924722ebbd938cf8e13d4b8d79cb027"}
Nov 03 00:05:02.282723 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.282683    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq" event={"ID":"9df7b833-c25f-48d0-8de2-401b05bd6796","Type":"ContainerStarted","Data":"d2f2f2c8c8be24f9b30c6f84d343156576b48cdd5efb5631820b64b69c02aacc"}
Nov 03 00:05:02.365635 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.365603    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:05:02.366379 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.366351    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:05:02.368691 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:02.368640    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq" podStartSLOduration=3.412096619 podStartE2EDuration="11.368624905s" totalImagesPullingTime="7.956528286s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:02.368319415 +0000 UTC m=+11.806494785" watchObservedRunningTime="2026-11-03 00:05:02.368624905 +0000 UTC m=+11.806800275"
Nov 03 00:05:03.064849 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:03.064807    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:03.065453 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:03.064951    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:03.287348 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:03.287299    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-6w7xg" event={"ID":"83c6f1f9-3e2f-4c6e-801f-29735ad355a2","Type":"ContainerStarted","Data":"2da48280eabd2f275de5a82c0b6467307f408247dd73012e1b1a92a795524546"}
Nov 03 00:05:03.323847 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:03.323755    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-6w7xg" podStartSLOduration=4.497922547 podStartE2EDuration="12.323741747s" totalImagesPullingTime="7.8258192s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:03.323601973 +0000 UTC m=+12.761777343" watchObservedRunningTime="2026-11-03 00:05:03.323741747 +0000 UTC m=+12.761917116"
Nov 03 00:05:04.064760 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:04.064715    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:04.064760 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:04.064739    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:04.065601 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:04.064828    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:04.065601 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:04.064977    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:04.292293 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:04.292209    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k25z9" event={"ID":"b24caade-1c7b-44e6-b04e-fbc0c44403fe","Type":"ContainerStarted","Data":"b2e32b879abb0cb40174aba522225501fd87fbaa5295659651ef5cab185f9359"}
Nov 03 00:05:04.295618 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:04.295583    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="f6e4bc304b0e67b8e715fd8580182648cde62bdbf4c083d9aca396091458776a" exitCode=0
Nov 03 00:05:04.295859 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:04.295830    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerDied","Data":"f6e4bc304b0e67b8e715fd8580182648cde62bdbf4c083d9aca396091458776a"}
Nov 03 00:05:04.369766 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:04.369662    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k25z9" podStartSLOduration=3.251558523 podStartE2EDuration="13.369647557s" totalImagesPullingTime="10.118089034s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:04.329192014 +0000 UTC m=+13.767367386" watchObservedRunningTime="2026-11-03 00:05:04.369647557 +0000 UTC m=+13.807822926"
Nov 03 00:05:05.064543 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:05.064509    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:05.064757 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:05.064728    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:06.065319 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:06.065077    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:06.065904 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:06.065155    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:06.065904 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:06.065410    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:06.065904 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:06.065576    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:07.064356 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:07.064324    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:07.064574 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:07.064444    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:07.304144 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:07.304110    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="11920084ab51b4af7187bc2e0b6813f4ebc66675292f759fb26eebf427a1dfd7" exitCode=0
Nov 03 00:05:07.304526 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:07.304165    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerDied","Data":"11920084ab51b4af7187bc2e0b6813f4ebc66675292f759fb26eebf427a1dfd7"}
Nov 03 00:05:08.064833 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.064812    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:08.064925 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.064847    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:08.064925 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.064916    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:08.065006 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.064980    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:08.309675 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.309510    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"5906257e4efd71c7b5759e5fe120848f7f6d39904d9681f7911dcb8233a12059"}
Nov 03 00:05:08.310037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.309717    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"8345626884d37534acff3acf6c2ded976e64eed1563562ca70fc8bc064aa30ae"}
Nov 03 00:05:08.310037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.309764    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"a75f5042c971d5ad7a7dd9a38b4119445ab776a7e4dca71c188f9058f1285803"}
Nov 03 00:05:08.310037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.309810    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"00efaf6dffe149db5903b9c5abb45174cac97761fd229da340491e48ae439350"}
Nov 03 00:05:08.310037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.309858    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"f513f989df92f3e8fcafb92374f7228acd7d3c467c769ed6f25d41322edddba1"}
Nov 03 00:05:08.310037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.309888    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"90a385e3277e0ed722d47a072256400e6d27021749ece2db054f8b7061c77559"}
Nov 03 00:05:08.313469 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.313438    2575 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="4763efdd5cc025115c2e2099db4cc46fde3fd1cfc032007ab5abd1af19043cff" exitCode=0
Nov 03 00:05:08.313569 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.313478    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerDied","Data":"4763efdd5cc025115c2e2099db4cc46fde3fd1cfc032007ab5abd1af19043cff"}
Nov 03 00:05:08.759635 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.759609    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:08.759790 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.759651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:08.759790 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.759743    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:05:08.759790 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.759743    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:05:08.759892 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.759796    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret podName:7ad1f241-5476-4a43-a616-f26eff05c43d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:24.759784196 +0000 UTC m=+34.197959542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret") pod "global-pull-secret-syncer-xhsst" (UID: "7ad1f241-5476-4a43-a616-f26eff05c43d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:05:08.759892 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.759810    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs podName:3ce18949-1971-4017-ab1c-5faa6d7412b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:24.75980353 +0000 UTC m=+34.197978878 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs") pod "network-metrics-daemon-wjrvj" (UID: "3ce18949-1971-4017-ab1c-5faa6d7412b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:05:08.860607 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:08.860578    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:08.860711 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.860699    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:05:08.860745 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.860715    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:05:08.860745 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.860724    2575 projected.go:196] Error preparing data for projected volume kube-api-access-8xp29 for pod openshift-network-diagnostics/network-check-target-8l8m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:05:08.860827 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:08.860765    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29 podName:04183b41-8425-43cd-a145-9e454673cad1 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:24.860754465 +0000 UTC m=+34.298929812 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8xp29" (UniqueName: "kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29") pod "network-check-target-8l8m5" (UID: "04183b41-8425-43cd-a145-9e454673cad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:05:09.065214 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:09.065133    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:09.065377 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:09.065299    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:09.318962 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:09.318880    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" event={"ID":"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2","Type":"ContainerStarted","Data":"53577ebcf4c04e5cba223e62c70dadc03092d8824514d7c51cba79c406f42e91"}
Nov 03 00:05:09.382715 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:09.382668    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bdnqz" podStartSLOduration=7.9429390699999995 podStartE2EDuration="18.382653537s" totalImagesPullingTime="10.439714467s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:09.370886291 +0000 UTC m=+18.809061660" watchObservedRunningTime="2026-11-03 00:05:09.382653537 +0000 UTC m=+18.820828906"
Nov 03 00:05:10.064347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:10.064310    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:10.064347 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:10.064345    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:10.064598 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:10.064440    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:10.064598 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:10.064552    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:11.065643 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:11.065613    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:11.066080 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:11.065703    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:11.327738 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:11.327662    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"acc5440fff6d8aff715fd1d23c523c3bffc7db51130d41c27f8e5163c03f2bcb"}
Nov 03 00:05:12.065143 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:12.065111    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:12.065143 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:12.065138    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:12.065332 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:12.065278    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:12.065385 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:12.065354    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:13.064471 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.064435    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:13.065136 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:13.064589    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:13.334977 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.334896    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" event={"ID":"1bf749d7-4d5c-4855-b945-fbf6b4953624","Type":"ContainerStarted","Data":"86ffbee3bd6e95e2026ce0f6f59eb18b91b5cae937defd65094756e04ffdb66a"}
Nov 03 00:05:13.335197 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.335098    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:05:13.335561 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.335536    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:05:13.335667 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.335636    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:05:13.352768 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.352740    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:05:13.353816 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.353794    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:05:13.413989 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:13.413944    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv" podStartSLOduration=7.768832152 podStartE2EDuration="22.413932052s" totalImagesPullingTime="14.6450999s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:13.402077125 +0000 UTC m=+22.840252494" watchObservedRunningTime="2026-11-03 00:05:13.413932052 +0000 UTC m=+22.852107420"
Nov 03 00:05:14.064964 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.064931    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:14.064964 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.064963    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:14.065444 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:14.065048    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:14.065444 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:14.065141    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:14.548217 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.547967    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhsst"]
Nov 03 00:05:14.548402 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.548225    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:14.548402 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:14.548337    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:14.548527 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.548509    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wjrvj"]
Nov 03 00:05:14.548631 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.548620    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:14.548737 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:14.548719    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:14.561875 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.561849    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8l8m5"]
Nov 03 00:05:14.562022 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:14.561919    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:14.562022 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:14.561980    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:16.064569 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:16.064534    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:16.064959 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:16.064534    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:16.064959 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:16.064652    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:16.064959 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:16.064666    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:16.064959 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:16.064750    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:16.064959 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:16.064805    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:18.064405 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:18.064369    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:18.064405 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:18.064390    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:18.064405 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:18.064389    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:18.064931 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:18.064524    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:18.064931 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:18.064567    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:18.064931 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:18.064615    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:18.354936 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:18.354836    2575 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9td9d_21ef2aeb-b54a-45f5-b58a-af731ebdfe11/konnectivity-agent/0.log" limit=1024
Nov 03 00:05:20.064507 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:20.064466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:20.064507 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:20.064514    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:20.064947 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:20.064568    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:20.064947 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:20.064587    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:20.064947 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:20.064639    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:20.064947 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:20.064722    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:21.147352 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:21.147323    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6lccb_cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0/dns-node-resolver/0.log"
Nov 03 00:05:22.064563 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:22.064516    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:22.064754 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:22.064565    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:22.064754 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:22.064629    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:22.064754 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:22.064676    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:22.064754 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:22.064674    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:22.064754 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:22.064728    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:22.546803 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:22.546737    2575 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wf7kp_fba3da01-3b60-48ec-8d2d-2d90478e334a/node-ca/0.log"
Nov 03 00:05:24.064783 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:24.064744    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:24.064783 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:24.064769    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:24.065224 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:24.064806    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:24.065224 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.064882    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:24.065224 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.064968    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:24.065224 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.065064    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:24.771735 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:24.771661    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:24.771735 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:24.771704    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:24.771905 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.771802    2575 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:05:24.771905 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.771815    2575 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 03 00:05:24.771905 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.771861    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs podName:3ce18949-1971-4017-ab1c-5faa6d7412b2 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:56.771845915 +0000 UTC m=+66.210021262 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs") pod "network-metrics-daemon-wjrvj" (UID: "3ce18949-1971-4017-ab1c-5faa6d7412b2") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 03 00:05:24.771905 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.771875    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret podName:7ad1f241-5476-4a43-a616-f26eff05c43d nodeName:}" failed. No retries permitted until 2026-11-03 00:05:56.771868883 +0000 UTC m=+66.210044229 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret") pod "global-pull-secret-syncer-xhsst" (UID: "7ad1f241-5476-4a43-a616-f26eff05c43d") : object "kube-system"/"original-pull-secret" not registered
Nov 03 00:05:24.872015 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:24.871978    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:24.872168 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.872111    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 03 00:05:24.872168 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.872125    2575 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 03 00:05:24.872168 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.872134    2575 projected.go:196] Error preparing data for projected volume kube-api-access-8xp29 for pod openshift-network-diagnostics/network-check-target-8l8m5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:05:24.872286 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:24.872186    2575 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29 podName:04183b41-8425-43cd-a145-9e454673cad1 nodeName:}" failed. No retries permitted until 2026-11-03 00:05:56.872170057 +0000 UTC m=+66.310345421 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8xp29" (UniqueName: "kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29") pod "network-check-target-8l8m5" (UID: "04183b41-8425-43cd-a145-9e454673cad1") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 03 00:05:26.064901 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:26.064872    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:26.065324 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:26.064874    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:26.065324 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:26.064970    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:26.065324 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:26.064873    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:26.065324 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:26.065038    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:26.065324 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:26.065113    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:28.064387 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:28.064352    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:28.064862 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:28.064429    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:28.064862 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:28.064454    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:28.064862 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:28.064557    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:28.064862 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:28.064588    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:28.064862 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:28.064653    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:30.064778 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:30.064746    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:30.064778 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:30.064775    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:30.065298 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:30.064842    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:30.065298 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:30.064857    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:30.065298 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:30.064910    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:30.065298 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:30.064977    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:32.064265 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:32.064232    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:32.064675 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:32.064332    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:32.064675 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:32.064355    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:32.064675 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:32.064423    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-xhsst" podUID="7ad1f241-5476-4a43-a616-f26eff05c43d"
Nov 03 00:05:32.064675 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:32.064476    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-wjrvj" podUID="3ce18949-1971-4017-ab1c-5faa6d7412b2"
Nov 03 00:05:32.064675 ip-10-0-136-97 kubenswrapper[2575]: E1103 00:05:32.064555    2575 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8l8m5" podUID="04183b41-8425-43cd-a145-9e454673cad1"
Nov 03 00:05:33.361723 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.361698    2575 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeReady"
Nov 03 00:05:33.362165 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.361828    2575 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 03 00:05:33.417197 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.417122    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rv656"]
Nov 03 00:05:33.436305 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.436280    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qc4b5"]
Nov 03 00:05:33.436426 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.436374    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.448420 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.448393    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9tx8l"]
Nov 03 00:05:33.448534 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.448494    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.450604 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.450587    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7b8cw\""
Nov 03 00:05:33.450721 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.450636    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:05:33.451047 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.451030    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:05:33.460461 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.460441    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rv656"]
Nov 03 00:05:33.460545 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.460527    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.464785 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.464119    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tx8l"]
Nov 03 00:05:33.471501 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.471482    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qc4b5"]
Nov 03 00:05:33.533707 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.533674    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-metrics-tls\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.533853 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.533736    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-config-volume\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.533853 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.533757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-tmp-dir\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.533853 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.533825    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckjgg\" (UniqueName: \"kubernetes.io/projected/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-kube-api-access-ckjgg\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.550891 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.550866    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:05:33.550891 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.550874    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:05:33.551053 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.550939    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7llq5\""
Nov 03 00:05:33.551053 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.550948    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:05:33.551053 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.550948    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:05:33.551053 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.550965    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:05:33.551053 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.551015    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:05:33.557978 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.557967    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:05:33.557978 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.557973    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ct6nh\""
Nov 03 00:05:33.634313 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634283    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fedc973-716d-4cdc-9b47-748582eba143-crio-socket\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.634427 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-metrics-tls\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.634427 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634353    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-cert\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.634601 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634576    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhx6r\" (UniqueName: \"kubernetes.io/projected/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-kube-api-access-nhx6r\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.634663 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634621    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-config-volume\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.634746 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634731    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-tmp-dir\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.634801 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634757    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fedc973-716d-4cdc-9b47-748582eba143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.634801 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634781    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjgg\" (UniqueName: \"kubernetes.io/projected/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-kube-api-access-ckjgg\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.634907 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634828    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fedc973-716d-4cdc-9b47-748582eba143-data-volume\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.634907 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634865    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzz28\" (UniqueName: \"kubernetes.io/projected/5fedc973-716d-4cdc-9b47-748582eba143-kube-api-access-vzz28\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.634907 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.634897    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fedc973-716d-4cdc-9b47-748582eba143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.635052 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.635015    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-tmp-dir\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.635234 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.635214    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-config-volume\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.638029 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.638014    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-metrics-tls\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.653481 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.653458    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjgg\" (UniqueName: \"kubernetes.io/projected/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-kube-api-access-ckjgg\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.735650 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.735577    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx6r\" (UniqueName: \"kubernetes.io/projected/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-kube-api-access-nhx6r\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.735650 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.735619    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fedc973-716d-4cdc-9b47-748582eba143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.735822 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.735746    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fedc973-716d-4cdc-9b47-748582eba143-data-volume\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.735822 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.735791    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzz28\" (UniqueName: \"kubernetes.io/projected/5fedc973-716d-4cdc-9b47-748582eba143-kube-api-access-vzz28\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.735924 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.735826    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fedc973-716d-4cdc-9b47-748582eba143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.735924 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.735851    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fedc973-716d-4cdc-9b47-748582eba143-crio-socket\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.735924 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.735883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-cert\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.736099 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.736074    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fedc973-716d-4cdc-9b47-748582eba143-data-volume\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.736150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.736121    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fedc973-716d-4cdc-9b47-748582eba143-crio-socket\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.736465 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.736410    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fedc973-716d-4cdc-9b47-748582eba143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.738029 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.738010    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fedc973-716d-4cdc-9b47-748582eba143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.738391 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.738372    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-cert\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.753457 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.753433    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzz28\" (UniqueName: \"kubernetes.io/projected/5fedc973-716d-4cdc-9b47-748582eba143-kube-api-access-vzz28\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:33.753567 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.753546    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx6r\" (UniqueName: \"kubernetes.io/projected/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-kube-api-access-nhx6r\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.772384 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.772363    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:33.792199 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.792177    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:05:33.832989 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:33.831464    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:05:34.064529 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.064466    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:34.064657 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.064544    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:34.064657 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.064569    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:34.072046 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.072021    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rv656"]
Nov 03 00:05:34.075996 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:34.075972    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c3c97a4_e610_4a34_9fe8_b5e754372cd2.slice/crio-df69d97fcb8d9ba5936c47cf0ad7c2be59314f358137217db2630f596eb99531 WatchSource:0}: Error finding container df69d97fcb8d9ba5936c47cf0ad7c2be59314f358137217db2630f596eb99531: Status 404 returned error can't find the container with id df69d97fcb8d9ba5936c47cf0ad7c2be59314f358137217db2630f596eb99531
Nov 03 00:05:34.078564 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.078544    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:05:34.078691 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.078586    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:05:34.078799 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.078777    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v79z\""
Nov 03 00:05:34.078870 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.078797    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wsqcw\""
Nov 03 00:05:34.078870 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.078799    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:05:34.078870 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.078822    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:05:34.079027 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.078873    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dlnxq\""
Nov 03 00:05:34.093526 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.093506    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9tx8l"]
Nov 03 00:05:34.097226 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:34.097204    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79ae98d8_e66e_4d26_ba5e_d3e0f0d9421f.slice/crio-5bcc239ed96c060607dda9a0190217573c63071fee14c4d87b4d079746759982 WatchSource:0}: Error finding container 5bcc239ed96c060607dda9a0190217573c63071fee14c4d87b4d079746759982: Status 404 returned error can't find the container with id 5bcc239ed96c060607dda9a0190217573c63071fee14c4d87b4d079746759982
Nov 03 00:05:34.133357 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.133336    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qc4b5"]
Nov 03 00:05:34.137152 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:34.137130    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fedc973_716d_4cdc_9b47_748582eba143.slice/crio-3734545cae631fb2268cec7be98b800814ff2152cf2f238e079a66897fe9af88 WatchSource:0}: Error finding container 3734545cae631fb2268cec7be98b800814ff2152cf2f238e079a66897fe9af88: Status 404 returned error can't find the container with id 3734545cae631fb2268cec7be98b800814ff2152cf2f238e079a66897fe9af88
Nov 03 00:05:34.386912 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.386877    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv656" event={"ID":"8c3c97a4-e610-4a34-9fe8-b5e754372cd2","Type":"ContainerStarted","Data":"df69d97fcb8d9ba5936c47cf0ad7c2be59314f358137217db2630f596eb99531"}
Nov 03 00:05:34.388359 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.388328    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qc4b5" event={"ID":"5fedc973-716d-4cdc-9b47-748582eba143","Type":"ContainerStarted","Data":"64b26af42ebc543070af927f1c8df0cb3beec14aef4033e8956225a1898961cd"}
Nov 03 00:05:34.388470 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.388381    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qc4b5" event={"ID":"5fedc973-716d-4cdc-9b47-748582eba143","Type":"ContainerStarted","Data":"3734545cae631fb2268cec7be98b800814ff2152cf2f238e079a66897fe9af88"}
Nov 03 00:05:34.389311 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:34.389291    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tx8l" event={"ID":"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f","Type":"ContainerStarted","Data":"5bcc239ed96c060607dda9a0190217573c63071fee14c4d87b4d079746759982"}
Nov 03 00:05:35.394457 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.394378    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qc4b5" event={"ID":"5fedc973-716d-4cdc-9b47-748582eba143","Type":"ContainerStarted","Data":"f04d55699955982c604b0a5678abbefd578f7a125b598687973a866411ae93c7"}
Nov 03 00:05:35.600037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.600008    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"]
Nov 03 00:05:35.603137 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.603113    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.634010 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.633847    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 00:05:35.634010 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.633853    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 00:05:35.634010 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.633961    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rrqd5\""
Nov 03 00:05:35.634010 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.633965    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 00:05:35.634479 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.634462    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 00:05:35.634593 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.634509    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 00:05:35.646558 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.646491    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"]
Nov 03 00:05:35.752311 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.752273    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.752499 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.752338    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlf4r\" (UniqueName: \"kubernetes.io/projected/5a1b5632-bc8c-4692-8e60-816ef5b35575-kube-api-access-rlf4r\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.752499 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.752425    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.752499 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.752455    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1b5632-bc8c-4692-8e60-816ef5b35575-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.853500 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.853463    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.853612 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.853510    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlf4r\" (UniqueName: \"kubernetes.io/projected/5a1b5632-bc8c-4692-8e60-816ef5b35575-kube-api-access-rlf4r\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.853612 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.853552    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.853612 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.853575    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1b5632-bc8c-4692-8e60-816ef5b35575-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.854188 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.854156    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1b5632-bc8c-4692-8e60-816ef5b35575-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.857047 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.857023    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.857047 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.857036    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.873520 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.873486    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlf4r\" (UniqueName: \"kubernetes.io/projected/5a1b5632-bc8c-4692-8e60-816ef5b35575-kube-api-access-rlf4r\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.935395 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.927381    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:05:35.935395 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.928907    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c6c5cc84b-9bx56"]
Nov 03 00:05:35.937792 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.936735    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:35.953833 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.953549    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6c5cc84b-9bx56"]
Nov 03 00:05:35.954100 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.954079    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 00:05:35.954184 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.954122    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5qw2l\""
Nov 03 00:05:35.955111 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.955081    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 00:05:35.963707 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.963266    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 00:05:35.963707 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.963395    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 00:05:35.963707 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.963473    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 00:05:35.963707 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.963562    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 00:05:35.963707 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:35.963640    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 00:05:36.055422 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.055390    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.055562 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.055457    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.055562 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.055524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.055713 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.055692    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.055778 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.055752    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.055831 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.055784    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.055831 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.055810    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wssr\" (UniqueName: \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.157173    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.157218    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.157320    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.157367    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.157395    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.157420    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wssr\" (UniqueName: \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.157455    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.158185    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.158498    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.158872    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.158957 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.158878    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.160506 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.160480    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.161878 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.161855    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.177464 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.177437    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wssr\" (UniqueName: \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.236474 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.236390    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"]
Nov 03 00:05:36.242218 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:36.242191    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a1b5632_bc8c_4692_8e60_816ef5b35575.slice/crio-c61005ae1dade31ef22d219da9c2fcd052c154a2a4550a29470734a200332130 WatchSource:0}: Error finding container c61005ae1dade31ef22d219da9c2fcd052c154a2a4550a29470734a200332130: Status 404 returned error can't find the container with id c61005ae1dade31ef22d219da9c2fcd052c154a2a4550a29470734a200332130
Nov 03 00:05:36.273007 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.272982    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:36.398944 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.398901    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz" event={"ID":"5a1b5632-bc8c-4692-8e60-816ef5b35575","Type":"ContainerStarted","Data":"c61005ae1dade31ef22d219da9c2fcd052c154a2a4550a29470734a200332130"}
Nov 03 00:05:36.401667 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.401642    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv656" event={"ID":"8c3c97a4-e610-4a34-9fe8-b5e754372cd2","Type":"ContainerStarted","Data":"5bee04595127391e0b39e17a98936d652c8eda40ed29e6cf534d083ab774f063"}
Nov 03 00:05:36.401794 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.401695    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rv656" event={"ID":"8c3c97a4-e610-4a34-9fe8-b5e754372cd2","Type":"ContainerStarted","Data":"a94718d9f073b0d02fb7403da87eac81fe9f11fce9ce580c8d8050ecdfb4c00e"}
Nov 03 00:05:36.401978 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.401954    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:36.440815 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.440719    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-rv656" podStartSLOduration=1.673309215 podStartE2EDuration="3.440697634s" totalImagesPullingTime="1.767388419s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:36.440056778 +0000 UTC m=+45.878232151" watchObservedRunningTime="2026-11-03 00:05:36.440697634 +0000 UTC m=+45.878873005"
Nov 03 00:05:36.577863 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:36.577835    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c6c5cc84b-9bx56"]
Nov 03 00:05:36.581664 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:36.581559    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ce8d43_d4b6_4b23_a235_624a0e30612d.slice/crio-46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6 WatchSource:0}: Error finding container 46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6: Status 404 returned error can't find the container with id 46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6
Nov 03 00:05:37.407370 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:37.407327    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qc4b5" event={"ID":"5fedc973-716d-4cdc-9b47-748582eba143","Type":"ContainerStarted","Data":"59e252742bf21a700af6e25e8d37f84a0a331fe7090623bc8a42712bfae49d93"}
Nov 03 00:05:37.410199 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:37.409744    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9tx8l" event={"ID":"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f","Type":"ContainerStarted","Data":"f3622bd98b6562dc439bdf661b22ddb86954e4e02f5696a8e302412d549ba32d"}
Nov 03 00:05:37.412112 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:37.412083    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c5cc84b-9bx56" event={"ID":"32ce8d43-d4b6-4b23-a235-624a0e30612d","Type":"ContainerStarted","Data":"46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6"}
Nov 03 00:05:37.447416 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:37.447368    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qc4b5" podStartSLOduration=2.145404852 podStartE2EDuration="4.447353008s" totalImagesPullingTime="2.301948156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:37.446858547 +0000 UTC m=+46.885033919" watchObservedRunningTime="2026-11-03 00:05:37.447353008 +0000 UTC m=+46.885528381"
Nov 03 00:05:37.482589 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:37.482532    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9tx8l" podStartSLOduration=1.614931362 podStartE2EDuration="4.482518664s" totalImagesPullingTime="2.867587302s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:37.48179223 +0000 UTC m=+46.919967598" watchObservedRunningTime="2026-11-03 00:05:37.482518664 +0000 UTC m=+46.920694034"
Nov 03 00:05:38.419691 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:38.419406    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz" event={"ID":"5a1b5632-bc8c-4692-8e60-816ef5b35575","Type":"ContainerStarted","Data":"694fbe46dd722abf20a005d8c385994f733c302909821c76e43386cb17d1f35c"}
Nov 03 00:05:38.419691 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:38.419513    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz" event={"ID":"5a1b5632-bc8c-4692-8e60-816ef5b35575","Type":"ContainerStarted","Data":"db0775e69c041495abc3c4f5ddb497dd14597829d3407ca62fba0961718c605f"}
Nov 03 00:05:38.455990 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:38.455940    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz" podStartSLOduration=2.201614354 podStartE2EDuration="3.455927583s" totalImagesPullingTime="1.254313229s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:38.454905183 +0000 UTC m=+47.893080552" watchObservedRunningTime="2026-11-03 00:05:38.455927583 +0000 UTC m=+47.894102951"
Nov 03 00:05:39.424609 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:39.424569    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c5cc84b-9bx56" event={"ID":"32ce8d43-d4b6-4b23-a235-624a0e30612d","Type":"ContainerStarted","Data":"1392cd07b34b5073656227d54c10341d224e0e7696da2d9d920a1b99376648b8"}
Nov 03 00:05:39.463921 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:39.463867    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6c6c5cc84b-9bx56" podStartSLOduration=1.791361213 podStartE2EDuration="4.463853236s" totalImagesPullingTime="2.672492023s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:39.462678395 +0000 UTC m=+48.900853763" watchObservedRunningTime="2026-11-03 00:05:39.463853236 +0000 UTC m=+48.902028604"
Nov 03 00:05:40.190442 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.190396    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"]
Nov 03 00:05:40.192508 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.192491    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.208562 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.208533    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 00:05:40.217904 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.217876    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vhndk\""
Nov 03 00:05:40.217980 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.217942    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 00:05:40.220949 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.220933    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"]
Nov 03 00:05:40.249829 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.249803    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"]
Nov 03 00:05:40.251919 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.251901    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.268158 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.268129    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ksp7h"]
Nov 03 00:05:40.270033 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.270016    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.277896 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.277758    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 00:05:40.279235 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.279208    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"]
Nov 03 00:05:40.295899 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.295880    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 00:05:40.298160 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298137    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5z47\" (UniqueName: \"kubernetes.io/projected/0da10585-07cb-473c-94c2-d9a8c519ea73-kube-api-access-v5z47\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298270 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298173    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-textfile\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298343 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298320    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-tls\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298400 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298351    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.298400 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298376    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n88np\" (UniqueName: \"kubernetes.io/projected/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-api-access-n88np\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.298501 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298407    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-root\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298501 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298431    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-accelerators-collector-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298501 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298484    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-wtmp\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298619 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298525    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-metrics-client-ca\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298682 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298665    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.298735 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298698    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.298787 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjv8x\" (UniqueName: \"kubernetes.io/projected/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-kube-api-access-mjv8x\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.298787 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298755    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-sys\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.298863 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298788    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.298905 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298887    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.298945 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298919    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.298945 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.298939    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.299037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.299018    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.299067 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.299049    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.308842 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.308820    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 00:05:40.308910 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.308858    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 00:05:40.308954 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.308940    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xdt4r\""
Nov 03 00:05:40.309010 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.308999    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 00:05:40.309050 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.309038    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 00:05:40.309103 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.309091    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56fpg\""
Nov 03 00:05:40.399883 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.399854    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-tls\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.399883 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.399883    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.399900    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88np\" (UniqueName: \"kubernetes.io/projected/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-api-access-n88np\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.399918    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-root\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.399935    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-accelerators-collector-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.399951    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-wtmp\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.399967    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-metrics-client-ca\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400027    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400030    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-root\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400052    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjv8x\" (UniqueName: \"kubernetes.io/projected/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-kube-api-access-mjv8x\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400093    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-sys\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400150 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400129    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.400717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400194    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.400717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400237    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.400717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400294    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.400717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400329    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400355    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.400717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400387    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5z47\" (UniqueName: \"kubernetes.io/projected/0da10585-07cb-473c-94c2-d9a8c519ea73-kube-api-access-v5z47\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.400717 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400421    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-textfile\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.401013 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400731    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-accelerators-collector-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.401013 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400899    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-metrics-client-ca\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.401013 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.400947    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-textfile\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.401605 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.401249    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.401605 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.401286    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.401605 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.401344    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-sys\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.401605 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.401439    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-wtmp\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.401858 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.401671    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.402186 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.402157    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.402874 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.402846    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-tls\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.403222 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.403200    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.403320 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.403288    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.403537 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.403518    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.404001 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.403980    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.404075 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.404012    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.419335 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.419314    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjv8x\" (UniqueName: \"kubernetes.io/projected/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-kube-api-access-mjv8x\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.419427 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.419343    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88np\" (UniqueName: \"kubernetes.io/projected/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-api-access-n88np\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.419676 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.419655    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5z47\" (UniqueName: \"kubernetes.io/projected/0da10585-07cb-473c-94c2-d9a8c519ea73-kube-api-access-v5z47\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.504151 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.504083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:05:40.564637 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.564610    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:05:40.583987 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.583962    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:05:40.594616 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:40.594582    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0da10585_07cb_473c_94c2_d9a8c519ea73.slice/crio-a9377decffd5c33c725c860453b1f61664e337161e17163c840c805ed4c06562 WatchSource:0}: Error finding container a9377decffd5c33c725c860453b1f61664e337161e17163c840c805ed4c06562: Status 404 returned error can't find the container with id a9377decffd5c33c725c860453b1f61664e337161e17163c840c805ed4c06562
Nov 03 00:05:40.806415 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.806350    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"]
Nov 03 00:05:40.809644 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:40.809622    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2de66e0_f1f5_44d4_8eca_3bd623d6fd84.slice/crio-8b30957b0c0d2bf95621ac39ec85e881a2df7d721382ce5da02a2a78db6eb649 WatchSource:0}: Error finding container 8b30957b0c0d2bf95621ac39ec85e881a2df7d721382ce5da02a2a78db6eb649: Status 404 returned error can't find the container with id 8b30957b0c0d2bf95621ac39ec85e881a2df7d721382ce5da02a2a78db6eb649
Nov 03 00:05:40.858545 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:40.858521    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"]
Nov 03 00:05:40.861975 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:40.861951    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcd7e2a0_6e98_4417_bbc9_25868b1ca911.slice/crio-c6a2b1bdef557c95ba59c06db8d52331ba8e867b9a9cd48a3a53c6c0dff7127e WatchSource:0}: Error finding container c6a2b1bdef557c95ba59c06db8d52331ba8e867b9a9cd48a3a53c6c0dff7127e: Status 404 returned error can't find the container with id c6a2b1bdef557c95ba59c06db8d52331ba8e867b9a9cd48a3a53c6c0dff7127e
Nov 03 00:05:41.432333 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.432299    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws" event={"ID":"dcd7e2a0-6e98-4417-bbc9-25868b1ca911","Type":"ContainerStarted","Data":"c6a2b1bdef557c95ba59c06db8d52331ba8e867b9a9cd48a3a53c6c0dff7127e"}
Nov 03 00:05:41.433716 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.433689    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ksp7h" event={"ID":"0da10585-07cb-473c-94c2-d9a8c519ea73","Type":"ContainerStarted","Data":"a9377decffd5c33c725c860453b1f61664e337161e17163c840c805ed4c06562"}
Nov 03 00:05:41.438459 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.438436    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs" event={"ID":"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84","Type":"ContainerStarted","Data":"2c3caf1b2dcd2c00181392471fdeccdb2b8ec39c5457575d646a0ba3ff8a7479"}
Nov 03 00:05:41.438616 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.438595    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs" event={"ID":"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84","Type":"ContainerStarted","Data":"a20e153c036d39c602ef17fb54306819b9e700de4613ff9a472697c1b0ecba0a"}
Nov 03 00:05:41.438678 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.438636    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs" event={"ID":"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84","Type":"ContainerStarted","Data":"8b30957b0c0d2bf95621ac39ec85e881a2df7d721382ce5da02a2a78db6eb649"}
Nov 03 00:05:41.689349 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.688439    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:05:41.691483 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.691450    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.706492 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706468    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 00:05:41.706686 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706669    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 00:05:41.706784 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706771    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 00:05:41.706847 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706766    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 00:05:41.706894 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706849    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 00:05:41.706894 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706887    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 00:05:41.706987 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706903    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 00:05:41.706987 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706905    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 00:05:41.706987 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.706961    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 00:05:41.707124 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.707021    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tbmwn\""
Nov 03 00:05:41.711340 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711315    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711454 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711358    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711454 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711386    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711569 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711473    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711569 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711524    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711665 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711577    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711665 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711602    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711665 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711625    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711665 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711649    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711665 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711666    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711814 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711699    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711814 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711717    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.711814 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.711732    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9xxv\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.721103 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.721078    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:05:41.812487 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812443    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812487 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812499    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812706 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812530    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9xxv\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812706 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812580    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812706 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812706 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812625    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812706 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812651    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812706 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812681    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812880 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812713    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812880 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812732    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812880 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812752    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812880 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812768    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.812880 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.812795    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.813758 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.813718    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.814335 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.814302    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.816327 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.815996    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.817164 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.816493    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.817164 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.817044    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.817164 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.817055    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.817164 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.817123    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.817511 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.817478    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.818143 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.818122    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.819026 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.818984    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.819532 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.819507    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.819532 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.819524    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:41.834996 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:41.834966    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9xxv\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:42.013232 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.013199    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:05:42.341741 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.341708    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:05:42.349412 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:42.349375    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8b2a49b_6679_40ce_91f0_91fcfd7c7118.slice/crio-bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433 WatchSource:0}: Error finding container bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433: Status 404 returned error can't find the container with id bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433
Nov 03 00:05:42.445095 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.445046    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws" event={"ID":"dcd7e2a0-6e98-4417-bbc9-25868b1ca911","Type":"ContainerStarted","Data":"e1e3aedf2670ddf481881709d6885aaaaac531cc07811553c03c0d1f03501219"}
Nov 03 00:05:42.445303 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.445116    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws" event={"ID":"dcd7e2a0-6e98-4417-bbc9-25868b1ca911","Type":"ContainerStarted","Data":"bbf70e8391a801b3813c1beeb2b97ba5d0ec12c18eaa2f7200e61e6788d8954e"}
Nov 03 00:05:42.445303 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.445144    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws" event={"ID":"dcd7e2a0-6e98-4417-bbc9-25868b1ca911","Type":"ContainerStarted","Data":"fef231136a8d99f89d118a88c708064cb47f92a157e1761338d9c075d834e1a2"}
Nov 03 00:05:42.447589 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.447558    2575 generic.go:356] "Generic (PLEG): container finished" podID="0da10585-07cb-473c-94c2-d9a8c519ea73" containerID="801b11fdc26932b3ddb6da304d6c4bf1cd8b6a982ded042d7157abb390e3240a" exitCode=0
Nov 03 00:05:42.447748 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.447658    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ksp7h" event={"ID":"0da10585-07cb-473c-94c2-d9a8c519ea73","Type":"ContainerDied","Data":"801b11fdc26932b3ddb6da304d6c4bf1cd8b6a982ded042d7157abb390e3240a"}
Nov 03 00:05:42.452941 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.452910    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs" event={"ID":"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84","Type":"ContainerStarted","Data":"1349fad76d0013216adce4097ae6cdd8cf944b9f5f4eeaba95eb4b251e2bb61c"}
Nov 03 00:05:42.455555 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.455492    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerStarted","Data":"bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433"}
Nov 03 00:05:42.490629 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.490603    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"]
Nov 03 00:05:42.495893 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.495836    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws" podStartSLOduration=1.160814527 podStartE2EDuration="2.495817094s" totalImagesPullingTime="1.335002567s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:42.495387906 +0000 UTC m=+51.933563273" watchObservedRunningTime="2026-11-03 00:05:42.495817094 +0000 UTC m=+51.933992465"
Nov 03 00:05:42.498330 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.496032    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.515413 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.515392    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 00:05:42.515612 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.515594    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 00:05:42.515666 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.515650    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 00:05:42.515715 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.515704    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 00:05:42.515819 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.515804    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w8ktr\""
Nov 03 00:05:42.517987 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.517971    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 00:05:42.519212 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.519192    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ed77f9-5e31-428a-988c-7811b60e299a-metrics-client-ca\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.519518    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.519547    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-grpc-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.519567    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpp5m\" (UniqueName: \"kubernetes.io/projected/03ed77f9-5e31-428a-988c-7811b60e299a-kube-api-access-vpp5m\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.519592    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.519626    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.519708    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.520073    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.520174 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.520112    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ed77f9-5e31-428a-988c-7811b60e299a-tmp\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.524929 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.524910    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9s6uds6316t24\""
Nov 03 00:05:42.541228 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.541206    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"]
Nov 03 00:05:42.549710 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.549655    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs" podStartSLOduration=1.488906824 podStartE2EDuration="2.549639329s" totalImagesPullingTime="1.060732505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:42.549092884 +0000 UTC m=+51.987268276" watchObservedRunningTime="2026-11-03 00:05:42.549639329 +0000 UTC m=+51.987814698"
Nov 03 00:05:42.620720 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.620687    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.620720 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.620720    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-grpc-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.620952 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.620737    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpp5m\" (UniqueName: \"kubernetes.io/projected/03ed77f9-5e31-428a-988c-7811b60e299a-kube-api-access-vpp5m\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.620952 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.620884    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.620952 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.620954    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.621118 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.621005    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.621207 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.621179    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.621828 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.621509    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ed77f9-5e31-428a-988c-7811b60e299a-tmp\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.621828 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.621582    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ed77f9-5e31-428a-988c-7811b60e299a-metrics-client-ca\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.622099 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.622053    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ed77f9-5e31-428a-988c-7811b60e299a-tmp\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.622592 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.622568    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ed77f9-5e31-428a-988c-7811b60e299a-metrics-client-ca\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.624185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.624079    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.624185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.624100    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.624185 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.624159    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-grpc-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.624338 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.624228    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.625236 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.625205    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.625344 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.625319    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.638656 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.638637    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpp5m\" (UniqueName: \"kubernetes.io/projected/03ed77f9-5e31-428a-988c-7811b60e299a-kube-api-access-vpp5m\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:42.825977 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:42.825894    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:43.153286 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:43.153226    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"]
Nov 03 00:05:43.157581 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:43.157548    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03ed77f9_5e31_428a_988c_7811b60e299a.slice/crio-05188748994e5aa0c19d7b7aa28777c3fe4f9e5ce6929e38759916fd66851a3d WatchSource:0}: Error finding container 05188748994e5aa0c19d7b7aa28777c3fe4f9e5ce6929e38759916fd66851a3d: Status 404 returned error can't find the container with id 05188748994e5aa0c19d7b7aa28777c3fe4f9e5ce6929e38759916fd66851a3d
Nov 03 00:05:43.466797 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:43.466718    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ksp7h" event={"ID":"0da10585-07cb-473c-94c2-d9a8c519ea73","Type":"ContainerStarted","Data":"843ed33516b59b0d5a8988474b619709573667965a921a8b265914c901d7d817"}
Nov 03 00:05:43.466797 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:43.466793    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ksp7h" event={"ID":"0da10585-07cb-473c-94c2-d9a8c519ea73","Type":"ContainerStarted","Data":"5b10750ab9d804a5cd16cac5cdaf24bedf1d6ec3142ac54d8e9a53db3d1caa78"}
Nov 03 00:05:43.468617 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:43.468585    2575 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6" exitCode=0
Nov 03 00:05:43.468730 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:43.468629    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6"}
Nov 03 00:05:43.470037 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:43.470011    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" event={"ID":"03ed77f9-5e31-428a-988c-7811b60e299a","Type":"ContainerStarted","Data":"05188748994e5aa0c19d7b7aa28777c3fe4f9e5ce6929e38759916fd66851a3d"}
Nov 03 00:05:43.516183 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:43.516137    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ksp7h" podStartSLOduration=2.595930469 podStartE2EDuration="3.516124326s" totalImagesPullingTime="829.541339ms" totalInitContainerRuntime="90.652518ms" isStatefulPod=true podCreationTimestamp="2026-11-03 00:05:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:43.515077656 +0000 UTC m=+52.953253024" watchObservedRunningTime="2026-11-03 00:05:43.516124326 +0000 UTC m=+52.954299695"
Nov 03 00:05:44.716471 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.716425    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-c65fccf8-rt7bz"]
Nov 03 00:05:44.719097 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.719073    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.733323 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.733298    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 00:05:44.745017 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.744991    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 00:05:44.745158 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.745145    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 00:05:44.745299 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.745286    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4gj81ut6di7s2\""
Nov 03 00:05:44.747051 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747027    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c65fccf8-rt7bz"]
Nov 03 00:05:44.747162 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747131    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747229    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gs4dg\""
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747574    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-metrics-server-audit-profiles\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747637    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-client-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747675    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2w2j6\" (UniqueName: \"kubernetes.io/projected/182c2fcb-792e-429f-9ab3-875803a3fd66-kube-api-access-2w2j6\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747723    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747744    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/182c2fcb-792e-429f-9ab3-875803a3fd66-audit-log\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747783    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-client-certs\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.747902 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.747805    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-tls\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.849286    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-client-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.849359    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2j6\" (UniqueName: \"kubernetes.io/projected/182c2fcb-792e-429f-9ab3-875803a3fd66-kube-api-access-2w2j6\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.849422    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.849451    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/182c2fcb-792e-429f-9ab3-875803a3fd66-audit-log\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.849512    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-client-certs\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.849537    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-tls\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.849563    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-metrics-server-audit-profiles\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.850699    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-metrics-server-audit-profiles\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.851684 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.851380    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/182c2fcb-792e-429f-9ab3-875803a3fd66-audit-log\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.852121 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.851762    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.853759 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.853732    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-client-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.854236 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.854211    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-client-certs\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.855360 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.855334    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-tls\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:44.875750 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:44.875724    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2j6\" (UniqueName: \"kubernetes.io/projected/182c2fcb-792e-429f-9ab3-875803a3fd66-kube-api-access-2w2j6\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:45.034310 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.034207    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:05:45.062751 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.061610    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"]
Nov 03 00:05:45.075335 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.075308    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.086054 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.086016    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"]
Nov 03 00:05:45.089411 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.089392    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j9n2q\""
Nov 03 00:05:45.089552 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.089458    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 00:05:45.152517 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.152475    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.152693 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.152559    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfgmh\" (UniqueName: \"kubernetes.io/projected/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-kube-api-access-lfgmh\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.253626 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.253602    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.253716 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.253655    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgmh\" (UniqueName: \"kubernetes.io/projected/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-kube-api-access-lfgmh\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.255857 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.255840    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.274031 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.272907    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgmh\" (UniqueName: \"kubernetes.io/projected/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-kube-api-access-lfgmh\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.356142 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.356114    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-c65fccf8-rt7bz"]
Nov 03 00:05:45.360217 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.359363    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:05:45.363156 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:45.363126    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod182c2fcb_792e_429f_9ab3_875803a3fd66.slice/crio-0834dacd29061203ceabf362cb927ce0ff242b9ba15067e37532738bf861b0a8 WatchSource:0}: Error finding container 0834dacd29061203ceabf362cb927ce0ff242b9ba15067e37532738bf861b0a8: Status 404 returned error can't find the container with id 0834dacd29061203ceabf362cb927ce0ff242b9ba15067e37532738bf861b0a8
Nov 03 00:05:45.399103 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.399083    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:45.509139 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.505803    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerStarted","Data":"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"}
Nov 03 00:05:45.509139 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.505868    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerStarted","Data":"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"}
Nov 03 00:05:45.509139 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.505895    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerStarted","Data":"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"}
Nov 03 00:05:45.509139 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.507805    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"]
Nov 03 00:05:45.514562 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.514533    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.515108 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.515063    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" event={"ID":"03ed77f9-5e31-428a-988c-7811b60e299a","Type":"ContainerStarted","Data":"d7c05f4b6ebe9a49a37b7cd53ef8c46ef58a9dc9a0560ddf2f9db2ffd8a8db4f"}
Nov 03 00:05:45.515200 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.515125    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" event={"ID":"03ed77f9-5e31-428a-988c-7811b60e299a","Type":"ContainerStarted","Data":"099998b0e5b5b313988830799995e3cfd61893dd8e27565ee1ead72bc589f7ef"}
Nov 03 00:05:45.515200 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.515168    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" event={"ID":"03ed77f9-5e31-428a-988c-7811b60e299a","Type":"ContainerStarted","Data":"3a2edc7da55c55643836f56039f90ce8a123bf4fb1ddb26eae3a1baf20a3f02e"}
Nov 03 00:05:45.517118 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.517061    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz" event={"ID":"182c2fcb-792e-429f-9ab3-875803a3fd66","Type":"ContainerStarted","Data":"0834dacd29061203ceabf362cb927ce0ff242b9ba15067e37532738bf861b0a8"}
Nov 03 00:05:45.529766 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.529746    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 00:05:45.529870 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.529851    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 00:05:45.530015 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.530002    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rgf75\""
Nov 03 00:05:45.530090 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.530077    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 00:05:45.530140 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.530098    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 00:05:45.530140 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.530127    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 00:05:45.542007 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.541768    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 00:05:45.543294 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.543277    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"]
Nov 03 00:05:45.557133 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557041    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.557133 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557095    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.557322 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557186    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zv5n\" (UniqueName: \"kubernetes.io/projected/50e89fdd-eb0a-44c4-b904-9e7955f88504-kube-api-access-7zv5n\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.557322 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557216    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-serving-certs-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.557322 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557306    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.557466 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557335    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-metrics-client-ca\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.557466 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557367    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.557466 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.557397    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-federate-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.658712 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658682    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-serving-certs-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.658863 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658734    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.658863 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658760    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-metrics-client-ca\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.658863 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658786    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.658863 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658812    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-federate-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.659028 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658881    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.659028 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658913    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.659028 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.658946    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zv5n\" (UniqueName: \"kubernetes.io/projected/50e89fdd-eb0a-44c4-b904-9e7955f88504-kube-api-access-7zv5n\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.659713 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.659648    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-metrics-client-ca\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.659990 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.659721    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-serving-certs-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.659990 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.659808    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.661439 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.661420    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-federate-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.661536 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.661520    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.661892 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.661870    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.661967 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.661907    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.677694 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.677666    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zv5n\" (UniqueName: \"kubernetes.io/projected/50e89fdd-eb0a-44c4-b904-9e7955f88504-kube-api-access-7zv5n\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:45.718440 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.718415    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"]
Nov 03 00:05:45.722043 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:45.722018    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c4f0ae1_8981_44e8_ac34_2ee16ee2fae6.slice/crio-e3f6dbc10315280d5abb5d028f58c06fcdd9e4e77f8c79c447637eb742329a65 WatchSource:0}: Error finding container e3f6dbc10315280d5abb5d028f58c06fcdd9e4e77f8c79c447637eb742329a65: Status 404 returned error can't find the container with id e3f6dbc10315280d5abb5d028f58c06fcdd9e4e77f8c79c447637eb742329a65
Nov 03 00:05:45.840602 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:45.840538    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:05:46.164561 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.164326    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"]
Nov 03 00:05:46.170976 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:46.170932    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50e89fdd_eb0a_44c4_b904_9e7955f88504.slice/crio-14f8751721384b200f6703a22e3650e461d96900636af43d6326b88a0d5e3d1a WatchSource:0}: Error finding container 14f8751721384b200f6703a22e3650e461d96900636af43d6326b88a0d5e3d1a: Status 404 returned error can't find the container with id 14f8751721384b200f6703a22e3650e461d96900636af43d6326b88a0d5e3d1a
Nov 03 00:05:46.275805 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.275772    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:46.275947 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.275837    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:46.282705 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.282428    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:46.414973 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.414894    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rv656"
Nov 03 00:05:46.522806 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.522639    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq" event={"ID":"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6","Type":"ContainerStarted","Data":"e3f6dbc10315280d5abb5d028f58c06fcdd9e4e77f8c79c447637eb742329a65"}
Nov 03 00:05:46.529892 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.529776    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerStarted","Data":"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"}
Nov 03 00:05:46.529892 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.529843    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerStarted","Data":"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"}
Nov 03 00:05:46.535619 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.535041    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv" event={"ID":"50e89fdd-eb0a-44c4-b904-9e7955f88504","Type":"ContainerStarted","Data":"14f8751721384b200f6703a22e3650e461d96900636af43d6326b88a0d5e3d1a"}
Nov 03 00:05:46.540625 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:46.540433    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:05:47.542998 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:47.542964    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerStarted","Data":"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"}
Nov 03 00:05:47.549019 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:47.548997    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" event={"ID":"03ed77f9-5e31-428a-988c-7811b60e299a","Type":"ContainerStarted","Data":"7856aff6256d1647ab0f1c2f2873cb25727f77b6cde274ce35a5fa7157c781a4"}
Nov 03 00:05:47.549126 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:47.549065    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:47.549126 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:47.549111    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" event={"ID":"03ed77f9-5e31-428a-988c-7811b60e299a","Type":"ContainerStarted","Data":"6e7a1b0eced12f35cc37e00f0f2f4259124803c78b697e6c6523bd97465d989a"}
Nov 03 00:05:47.549228 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:47.549152    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" event={"ID":"03ed77f9-5e31-428a-988c-7811b60e299a","Type":"ContainerStarted","Data":"518febb580a2986b5e8baee1efb3f39524fb6a72c8c7d4bfeb44bfc1a2042c69"}
Nov 03 00:05:47.611728 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:47.611669    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.02429979 podStartE2EDuration="6.61164929s" totalImagesPullingTime="3.575131313s" totalInitContainerRuntime="12.218187ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:41 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:47.599506985 +0000 UTC m=+57.037682354" watchObservedRunningTime="2026-11-03 00:05:47.61164929 +0000 UTC m=+57.049824660"
Nov 03 00:05:47.644453 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:47.644389    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf" podStartSLOduration=2.573896842 podStartE2EDuration="5.644370493s" totalImagesPullingTime="3.070473651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:47.64241997 +0000 UTC m=+57.080595337" watchObservedRunningTime="2026-11-03 00:05:47.644370493 +0000 UTC m=+57.082545863"
Nov 03 00:05:48.553530 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.553433    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv" event={"ID":"50e89fdd-eb0a-44c4-b904-9e7955f88504","Type":"ContainerStarted","Data":"3fbe53d4f6e252502f8865d96f9845883f401bd1cb3529f2ff0c9c5f559fa3af"}
Nov 03 00:05:48.553950 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.553552    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv" event={"ID":"50e89fdd-eb0a-44c4-b904-9e7955f88504","Type":"ContainerStarted","Data":"80815f9dc615afa6a12e438b2fa6d83fbf056f20c2110767ae70363dc6673a3e"}
Nov 03 00:05:48.553950 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.553588    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv" event={"ID":"50e89fdd-eb0a-44c4-b904-9e7955f88504","Type":"ContainerStarted","Data":"64d0719a0d8c42125b52745f2ed926fde1e2a0e49bfdea456377bcadb2a4740a"}
Nov 03 00:05:48.555112 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.555086    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq" event={"ID":"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6","Type":"ContainerStarted","Data":"8f543b2c1cce7f37aa253ea61621818e53e075fda98c5228032c80c330b7bd74"}
Nov 03 00:05:48.555289 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.555241    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:48.556776 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.556726    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz" event={"ID":"182c2fcb-792e-429f-9ab3-875803a3fd66","Type":"ContainerStarted","Data":"4822a4078beaeaaad853ad08d145fc6299827321b98ed8787af3193351ab2abe"}
Nov 03 00:05:48.560514 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.560498    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:05:48.595040 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.595000    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv" podStartSLOduration=1.55438857 podStartE2EDuration="3.594988705s" totalImagesPullingTime="2.040600135s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:48.593642825 +0000 UTC m=+58.031818195" watchObservedRunningTime="2026-11-03 00:05:48.594988705 +0000 UTC m=+58.033164088"
Nov 03 00:05:48.631009 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.630971    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz" podStartSLOduration=2.534943089 podStartE2EDuration="4.630959838s" totalImagesPullingTime="2.096016749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:48.630211816 +0000 UTC m=+58.068387189" watchObservedRunningTime="2026-11-03 00:05:48.630959838 +0000 UTC m=+58.069135207"
Nov 03 00:05:48.667706 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.667662    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq" podStartSLOduration=1.926612425 podStartE2EDuration="3.667649948s" totalImagesPullingTime="1.741037523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:48.66621261 +0000 UTC m=+58.104387979" watchObservedRunningTime="2026-11-03 00:05:48.667649948 +0000 UTC m=+58.105825317"
Nov 03 00:05:48.871583 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.871549    2575 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5fcfc8dbfc-dspqk"]
Nov 03 00:05:48.875273 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.875235    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.901791 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.901765    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 00:05:48.903162 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.903145    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fcfc8dbfc-dspqk"]
Nov 03 00:05:48.996459 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996430    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.996459 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996462    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.996633 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996490    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrnc\" (UniqueName: \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.996633 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996622    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.996698 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996658    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.996698 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996677    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.996826 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996808    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:48.996866 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:48.996852    2575 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098090 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098061    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098230 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098099    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098230 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098148    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098230 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098170    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098230 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098200    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrnc\" (UniqueName: \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098485 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098246    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098485 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098307    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098485 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098336    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.098625 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.098544    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.099083 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.099062    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.099179 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.099126    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.099179 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.099132    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.099485 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.099468    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.101234 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.101210    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.101343 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.101317    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.116448 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.116423    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrnc\" (UniqueName: \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.186473 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.186406    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:49.481617 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.481429    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5fcfc8dbfc-dspqk"]
Nov 03 00:05:49.484610 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:49.484287    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d1799f5_bac5_4c4c_b399_6f1a5130e140.slice/crio-513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74 WatchSource:0}: Error finding container 513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74: Status 404 returned error can't find the container with id 513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74
Nov 03 00:05:49.564332 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.564300    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fcfc8dbfc-dspqk" event={"ID":"1d1799f5-bac5-4c4c-b399-6f1a5130e140","Type":"ContainerStarted","Data":"ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b"}
Nov 03 00:05:49.564818 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.564774    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fcfc8dbfc-dspqk" event={"ID":"1d1799f5-bac5-4c4c-b399-6f1a5130e140","Type":"ContainerStarted","Data":"513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74"}
Nov 03 00:05:49.604227 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:49.604165    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5fcfc8dbfc-dspqk" podStartSLOduration=1.60415194 podStartE2EDuration="1.60415194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:49.603273678 +0000 UTC m=+59.041449043" watchObservedRunningTime="2026-11-03 00:05:49.60415194 +0000 UTC m=+59.042327308"
Nov 03 00:05:53.563899 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:53.563873    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:05:56.773123 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.773094    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:56.773541 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.773157    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:56.797887 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.797852    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:05:56.798002 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.797897    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:05:56.806153 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.806133    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:56.806304 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.806280    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:56.874007 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.873977    2575 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:56.887043 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.887022    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:05:56.896875 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.896851    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v79z\""
Nov 03 00:05:56.900817 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.900803    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dlnxq\""
Nov 03 00:05:56.904943 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.904928    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:05:56.906862 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.906846    2575 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:05:56.908325 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.908301    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:05:56.917633 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:56.917613    2575 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:57.189391 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.189364    2575 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wsqcw\""
Nov 03 00:05:57.197433 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.197408    2575 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:05:57.206460 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.206321    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-wjrvj"]
Nov 03 00:05:57.211342 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:57.211317    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce18949_1971_4017_ab1c_5faa6d7412b2.slice/crio-2d558cc916bb56bf77c9928148473721927dae5a98a09e776641e31a271781ce WatchSource:0}: Error finding container 2d558cc916bb56bf77c9928148473721927dae5a98a09e776641e31a271781ce: Status 404 returned error can't find the container with id 2d558cc916bb56bf77c9928148473721927dae5a98a09e776641e31a271781ce
Nov 03 00:05:57.217516 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.217496    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-xhsst"]
Nov 03 00:05:57.221094 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:57.221069    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad1f241_5476_4a43_a616_f26eff05c43d.slice/crio-27a4859046975425fd6db3d2a4a8be7ccd62dd628a0f6c53c738c0e7b935e0a0 WatchSource:0}: Error finding container 27a4859046975425fd6db3d2a4a8be7ccd62dd628a0f6c53c738c0e7b935e0a0: Status 404 returned error can't find the container with id 27a4859046975425fd6db3d2a4a8be7ccd62dd628a0f6c53c738c0e7b935e0a0
Nov 03 00:05:57.495772 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.495688    2575 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8l8m5"]
Nov 03 00:05:57.498732 ip-10-0-136-97 kubenswrapper[2575]: W1103 00:05:57.498705    2575 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04183b41_8425_43cd_a145_9e454673cad1.slice/crio-438f3006e3072384162919c0b3715eb3762d984664f2cf359f07dc4b50ea767f WatchSource:0}: Error finding container 438f3006e3072384162919c0b3715eb3762d984664f2cf359f07dc4b50ea767f: Status 404 returned error can't find the container with id 438f3006e3072384162919c0b3715eb3762d984664f2cf359f07dc4b50ea767f
Nov 03 00:05:57.600042 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.600009    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-xhsst" event={"ID":"7ad1f241-5476-4a43-a616-f26eff05c43d","Type":"ContainerStarted","Data":"27a4859046975425fd6db3d2a4a8be7ccd62dd628a0f6c53c738c0e7b935e0a0"}
Nov 03 00:05:57.601197 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.601168    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8l8m5" event={"ID":"04183b41-8425-43cd-a145-9e454673cad1","Type":"ContainerStarted","Data":"438f3006e3072384162919c0b3715eb3762d984664f2cf359f07dc4b50ea767f"}
Nov 03 00:05:57.602326 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:57.602306    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wjrvj" event={"ID":"3ce18949-1971-4017-ab1c-5faa6d7412b2","Type":"ContainerStarted","Data":"2d558cc916bb56bf77c9928148473721927dae5a98a09e776641e31a271781ce"}
Nov 03 00:05:58.610672 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:58.610397    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wjrvj" event={"ID":"3ce18949-1971-4017-ab1c-5faa6d7412b2","Type":"ContainerStarted","Data":"17e8288d4c8ae4f7150aab9080a052c4d131dd8d16d88abb0df4870e35f81c32"}
Nov 03 00:05:58.610672 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:58.610462    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-wjrvj" event={"ID":"3ce18949-1971-4017-ab1c-5faa6d7412b2","Type":"ContainerStarted","Data":"d2a1b330fe5cbf230a9e405c49745863bdcd1279820a8080a8f3210f7fbf7a60"}
Nov 03 00:05:58.651186 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:58.651112    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-wjrvj" podStartSLOduration=66.606172919 podStartE2EDuration="1m7.651091877s" totalImagesPullingTime="1.044918958s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:05:58.648599002 +0000 UTC m=+68.086774371" watchObservedRunningTime="2026-11-03 00:05:58.651091877 +0000 UTC m=+68.089267241"
Nov 03 00:05:59.187538 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:59.187502    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:59.187710 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:59.187578    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:59.194410 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:59.194384    2575 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:59.620128 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:59.620089    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:05:59.689158 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:05:59.688477    2575 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6c5cc84b-9bx56"]
Nov 03 00:06:01.627130 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:06:01.627081    2575 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8l8m5" event={"ID":"04183b41-8425-43cd-a145-9e454673cad1","Type":"ContainerStarted","Data":"c2d1669f433d227dfa47a698de0ba71843b013523e46877e18203301cebea9c6"}
Nov 03 00:06:01.627586 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:06:01.627219    2575 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:06:01.664447 ip-10-0-136-97 kubenswrapper[2575]: I1103 00:06:01.664387    2575 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8l8m5" podStartSLOduration=67.417968555 podStartE2EDuration="1m10.664368107s" totalImagesPullingTime="3.246399552s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:04:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:01.6642202 +0000 UTC m=+71.102395572" watchObservedRunningTime="2026-11-03 00:06:01.664368107 +0000 UTC m=+71.102543482"
Nov 03 00:06:02.388019 ip-10-0-136-97 systemd[1]: Stopping Kubernetes Kubelet...
Nov 03 00:06:02.395867 ip-10-0-136-97 systemd[1]: kubelet.service: Deactivated successfully.
Nov 03 00:06:02.396035 ip-10-0-136-97 systemd[1]: Stopped Kubernetes Kubelet.
Nov 03 00:06:02.396083 ip-10-0-136-97 systemd[1]: kubelet.service: Consumed 4.859s CPU time.
Nov 03 00:06:02.410751 ip-10-0-136-97 systemd[1]: Starting Kubernetes Kubelet...
Nov 03 00:06:02.477234 ip-10-0-136-97 kubenswrapper[6000]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:06:02.477234 ip-10-0-136-97 kubenswrapper[6000]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 03 00:06:02.477234 ip-10-0-136-97 kubenswrapper[6000]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:06:02.477234 ip-10-0-136-97 kubenswrapper[6000]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 03 00:06:02.479335 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479223    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.479335 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479335    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.479335 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479339    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479343    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479346    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479349    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479352    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479354    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479357    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479360    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479362    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479365    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479367    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479370    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479375    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479379    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479382    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479385    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479387    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479394    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479397    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479400    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.479438 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479403    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479406    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479409    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479412    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479414    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479418    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479421    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479424    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479426    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479429    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479432    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479434    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479437    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479440    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479442    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479445    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479448    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479451    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479454    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.479913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479457    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479459    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479462    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479464    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479467    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479471    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479473    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479476    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479479    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479482    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479484    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479487    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479489    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479492    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479494    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479497    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479500    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479502    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479505    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.480401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479507    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479510    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479512    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479515    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479517    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479520    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479523    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479525    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479528    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479530    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479533    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479535    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479538    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479541    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479543    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479546    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479548    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479550    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479553    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.480858 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479556    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479558    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479561    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479564    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479566    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479569    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479571    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479597    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479601    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479605    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479609    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479612    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479615    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479618    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479621    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479624    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479626    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479629    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479631    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479634    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.481330 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479636    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479639    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479641    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479643    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479646    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479648    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.479652    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480433    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480440    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480443    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480446    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480448    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480451    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480454    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480457    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480460    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480462    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480465    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480468    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.481802 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480470    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480473    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480475    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480478    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480480    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480483    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480486    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480489    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480491    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480494    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480496    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480499    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480502    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480504    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480506    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480509    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480511    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480513    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480516    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480518    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.482318 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480522    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480525    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480532    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480535    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480537    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480540    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480542    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480545    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480547    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480549    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480552    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480555    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480557    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480560    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480562    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480565    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480568    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480571    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480573    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.482803 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480576    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480578    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480581    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480583    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480586    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480588    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480591    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480593    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480596    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480598    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480601    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480603    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480605    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480608    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480611    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480615    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480617    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480620    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480623    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480625    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.483289 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480628    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480631    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480635    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480638    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480641    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480645    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480648    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480651    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480654    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480658    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480661    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480664    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480667    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480669    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480671    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480674    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480676    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480679    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480681    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.483783 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480684    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480686    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480688    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480691    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480693    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480696    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480699    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480701    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480710    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480713    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480716    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480718    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480721    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480723    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480725    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.480728    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481214    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481219    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481222    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481225    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.484272 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481228    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481230    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481233    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481235    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481240    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481244    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481247    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481268    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481271    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481274    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481276    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481279    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481282    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481284    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481287    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481289    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481292    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481294    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481297    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481299    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.484776 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481303    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481306    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481309    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481311    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481314    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481316    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481319    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481322    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481324    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481327    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481329    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481332    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481334    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481337    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481340    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481342    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481345    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481348    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481350    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.485358 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481353    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481355    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481359    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481362    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481365    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481368    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481370    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481373    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481375    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481378    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481380    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481383    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481385    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481388    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481392    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481394    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481397    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481399    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481401    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.486182 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481404    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481406    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481409    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481412    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481414    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481417    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481420    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481422    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481425    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481427    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481430    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481432    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481434    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481437    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481439    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481442    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481444    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481447    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481449    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481451    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.487033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481454    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481456    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481459    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481461    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481463    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481466    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481469    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481471    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481474    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481477    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481479    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481482    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481484    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481486    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481489    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481491    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481495    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481497    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481499    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481502    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.487908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481504    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481506    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481509    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.481511    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481601    6000 flags.go:64] FLAG: --address="0.0.0.0"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481609    6000 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481616    6000 flags.go:64] FLAG: --anonymous-auth="true"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481623    6000 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481632    6000 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481636    6000 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481640    6000 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481644    6000 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481648    6000 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481651    6000 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481654    6000 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481658    6000 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481661    6000 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481664    6000 flags.go:64] FLAG: --cgroup-root=""
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481667    6000 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481670    6000 flags.go:64] FLAG: --client-ca-file=""
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481672    6000 flags.go:64] FLAG: --cloud-provider="external"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481676    6000 flags.go:64] FLAG: --cluster-dns="[]"
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481685    6000 flags.go:64] FLAG: --cluster-domain=""
Nov 03 00:06:02.488922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481688    6000 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481691    6000 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481694    6000 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481697    6000 flags.go:64] FLAG: --container-log-max-files="5"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481702    6000 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481705    6000 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481709    6000 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481712    6000 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481715    6000 flags.go:64] FLAG: --contention-profiling="false"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481718    6000 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481721    6000 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481724    6000 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481726    6000 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481730    6000 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481733    6000 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481736    6000 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481739    6000 flags.go:64] FLAG: --enable-load-reader="false"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481745    6000 flags.go:64] FLAG: --enable-server="true"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481748    6000 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481752    6000 flags.go:64] FLAG: --event-burst="100"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481755    6000 flags.go:64] FLAG: --event-qps="50"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481758    6000 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481761    6000 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481764    6000 flags.go:64] FLAG: --eviction-hard=""
Nov 03 00:06:02.489712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481768    6000 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481771    6000 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481774    6000 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481777    6000 flags.go:64] FLAG: --eviction-soft=""
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481780    6000 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481783    6000 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481785    6000 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481789    6000 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481792    6000 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481795    6000 flags.go:64] FLAG: --fail-swap-on="true"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481797    6000 flags.go:64] FLAG: --feature-gates=""
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481804    6000 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481808    6000 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481812    6000 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481815    6000 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481818    6000 flags.go:64] FLAG: --healthz-port="10248"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481821    6000 flags.go:64] FLAG: --help="false"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481824    6000 flags.go:64] FLAG: --hostname-override="ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481827    6000 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481830    6000 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481833    6000 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481836    6000 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481840    6000 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481843    6000 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 03 00:06:02.490316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481846    6000 flags.go:64] FLAG: --image-service-endpoint=""
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481850    6000 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481853    6000 flags.go:64] FLAG: --kube-api-burst="100"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481855    6000 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481858    6000 flags.go:64] FLAG: --kube-api-qps="50"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481861    6000 flags.go:64] FLAG: --kube-reserved=""
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481864    6000 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481867    6000 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481870    6000 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481872    6000 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481875    6000 flags.go:64] FLAG: --lock-file=""
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481878    6000 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481881    6000 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481884    6000 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481889    6000 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481892    6000 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481895    6000 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481898    6000 flags.go:64] FLAG: --logging-format="text"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481901    6000 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481904    6000 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481908    6000 flags.go:64] FLAG: --manifest-url=""
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481911    6000 flags.go:64] FLAG: --manifest-url-header=""
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481916    6000 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481919    6000 flags.go:64] FLAG: --max-open-files="1000000"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481922    6000 flags.go:64] FLAG: --max-pods="110"
Nov 03 00:06:02.490912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481925    6000 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481928    6000 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481931    6000 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481934    6000 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481937    6000 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481940    6000 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481942    6000 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481949    6000 flags.go:64] FLAG: --node-status-max-images="50"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481954    6000 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481957    6000 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481960    6000 flags.go:64] FLAG: --pod-cidr=""
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481963    6000 flags.go:64] FLAG: --pod-manifest-path=""
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481966    6000 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481969    6000 flags.go:64] FLAG: --pods-per-core="0"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481972    6000 flags.go:64] FLAG: --port="10250"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481975    6000 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481978    6000 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c6b3e2241db61f1b"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481981    6000 flags.go:64] FLAG: --qos-reserved=""
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481984    6000 flags.go:64] FLAG: --read-only-port="10255"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481987    6000 flags.go:64] FLAG: --register-node="true"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481990    6000 flags.go:64] FLAG: --register-with-taints=""
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481994    6000 flags.go:64] FLAG: --registry-burst="10"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481996    6000 flags.go:64] FLAG: --registry-qps="5"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.481999    6000 flags.go:64] FLAG: --reserved-cpus=""
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482005    6000 flags.go:64] FLAG: --reserved-memory=""
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482009    6000 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 03 00:06:02.491598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482012    6000 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482016    6000 flags.go:64] FLAG: --rotate-certificates="false"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482019    6000 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482022    6000 flags.go:64] FLAG: --runonce="false"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482025    6000 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482027    6000 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482030    6000 flags.go:64] FLAG: --seccomp-default="false"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482033    6000 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482036    6000 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482039    6000 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482042    6000 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482045    6000 flags.go:64] FLAG: --storage-driver-password="root"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482048    6000 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482051    6000 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482055    6000 flags.go:64] FLAG: --storage-driver-user="root"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482058    6000 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482061    6000 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482064    6000 flags.go:64] FLAG: --system-cgroups=""
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482067    6000 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482072    6000 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482075    6000 flags.go:64] FLAG: --tls-cert-file=""
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482078    6000 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482082    6000 flags.go:64] FLAG: --tls-min-version=""
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482085    6000 flags.go:64] FLAG: --tls-private-key-file=""
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482088    6000 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 03 00:06:02.492233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482090    6000 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482093    6000 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482096    6000 flags.go:64] FLAG: --v="2"
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482100    6000 flags.go:64] FLAG: --version="false"
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482105    6000 flags.go:64] FLAG: --vmodule=""
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482108    6000 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482112    6000 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482244    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482249    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482270    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482273    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482276    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482279    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482281    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482284    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482286    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482289    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482292    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482295    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482297    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482300    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.492866 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482304    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482307    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482309    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482312    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482314    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482317    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482320    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482324    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482328    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482331    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482335    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482337    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482341    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482343    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482346    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482348    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482351    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482353    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482356    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.493549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482361    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482363    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482366    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482368    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482371    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482373    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482376    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482378    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482381    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482383    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482386    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482388    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482391    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482396    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482399    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482403    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482406    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482408    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482411    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.494091 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482413    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482416    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482418    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482421    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482423    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482426    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482428    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482431    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482433    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482435    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482438    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482440    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482443    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482447    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482450    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482452    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482455    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482457    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482460    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482462    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.494688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482465    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482467    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482470    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482472    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482475    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482477    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482481    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482483    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482486    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482488    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482491    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482493    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482496    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482498    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482500    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482503    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482505    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482508    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482511    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482513    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.495276 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482516    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482518    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482521    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482523    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482526    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482529    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482532    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482535    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482537    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482540    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482542    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482545    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482547    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482550    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.482575    6000 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:06:02.495816 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482611    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482618    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482622    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482627    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482630    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482633    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482637    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482641    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482644    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482647    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482650    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482653    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482655    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482658    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482660    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482662    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482665    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482667    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482670    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.496194 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482673    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482675    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482677    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482680    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482700    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482705    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482709    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482712    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482714    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482717    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482720    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482723    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482725    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482727    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482730    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482732    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482736    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482739    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482741    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482744    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.496818 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482747    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482749    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482752    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482754    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482757    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482759    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482762    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482764    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482767    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482769    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482772    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482774    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482777    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482779    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482783    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482785    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482789    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482792    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482795    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482797    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.497365 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482800    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482802    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482805    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482807    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482810    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482813    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482815    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482818    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482822    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482824    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482827    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482829    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482831    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482834    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482836    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482839    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482841    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482844    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482847    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.497903 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482849    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482851    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482854    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482856    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482859    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482861    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482863    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482866    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482869    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482874    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482887    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482890    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482893    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482897    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482900    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482902    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482905    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482907    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482910    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.498450 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482912    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482915    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482920    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482922    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482925    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482927    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482929    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482932    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.482935    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.487551    6000 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.487571    6000 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.487594    6000 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.487602    6000 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487685    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487692    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487697    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487701    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487706    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487710    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487714    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.498991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487719    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487723    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487727    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487731    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487736    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487740    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487744    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487748    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487752    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487757    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487761    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487765    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487769    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487773    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487777    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487781    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487786    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487790    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487795    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487799    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.499633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487803    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487807    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487854    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487958    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487985    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.487996    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488005    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488014    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488022    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488035    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488047    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488057    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488066    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488074    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488082    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488097    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488104    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488112    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488118    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.500185 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488125    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488132    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488139    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488147    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488155    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488163    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488170    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488178    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488186    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488691    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488711    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488717    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488723    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488728    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488733    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488737    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488741    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488746    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488750    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.500710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488755    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488760    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488770    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488775    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488783    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488789    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488796    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488800    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488805    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488809    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488814    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488818    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488822    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488826    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488830    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488840    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488844    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488848    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488852    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.501283 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488856    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488860    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488865    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488869    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488874    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488878    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488883    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488886    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488896    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488900    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488903    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488908    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488912    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488915    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488919    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488924    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488928    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488933    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488937    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488942    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.501780 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488951    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.488955    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.489185    6000 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489374    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489380    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489384    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489387    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489390    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489393    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489396    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489398    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489400    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489403    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489405    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489408    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489410    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.502285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489413    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489415    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489418    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489422    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489426    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489429    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489431    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489434    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489436    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489439    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489441    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489444    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489448    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489451    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489453    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489456    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489458    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489461    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489463    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.502669 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489466    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489469    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489471    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489475    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489477    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489480    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489482    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489484    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489487    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489490    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489492    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489496    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489500    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489503    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489505    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489508    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489510    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489513    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489515    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489517    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.503216 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489520    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489522    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489526    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489529    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489531    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489534    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489536    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489539    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489541    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489544    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489546    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489549    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489551    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489553    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489556    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489559    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489561    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489564    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489566    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.503914 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489569    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489571    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489574    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489576    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489578    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489581    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489583    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489586    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489588    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489590    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489593    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489595    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489598    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489600    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489603    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489606    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489608    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489611    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489613    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489615    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.504600 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489618    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489621    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489623    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489626    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489628    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489631    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489633    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489636    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489638    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489660    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489664    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489667    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489670    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489674    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489677    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.489703    6000 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 03 00:06:02.505081 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489736    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489740    6000 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489743    6000 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489746    6000 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489748    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489751    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489753    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489756    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489758    6000 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489761    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489763    6000 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489766    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489769    6000 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489773    6000 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489776    6000 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489779    6000 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489781    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489784    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489786    6000 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 03 00:06:02.505569 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489789    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489791    6000 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489793    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489796    6000 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489799    6000 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489801    6000 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489804    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489806    6000 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489809    6000 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489811    6000 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489814    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489817    6000 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489820    6000 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489823    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489825    6000 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489827    6000 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489830    6000 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489833    6000 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489835    6000 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489838    6000 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 03 00:06:02.506068 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489841    6000 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489845    6000 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489847    6000 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489850    6000 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489852    6000 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489855    6000 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489858    6000 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489860    6000 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489863    6000 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489865    6000 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489867    6000 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489870    6000 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489872    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489875    6000 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489878    6000 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489880    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489883    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489885    6000 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489887    6000 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489890    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 03 00:06:02.506571 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489892    6000 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489895    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489897    6000 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489899    6000 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489902    6000 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489905    6000 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489907    6000 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489910    6000 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489912    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489915    6000 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489918    6000 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489921    6000 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489923    6000 feature_gate.go:386] unrecognized feature gate: Example2
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489925    6000 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489928    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489930    6000 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489933    6000 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489935    6000 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489937    6000 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489940    6000 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 03 00:06:02.507120 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489942    6000 feature_gate.go:386] unrecognized feature gate: Example
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489944    6000 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489947    6000 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489950    6000 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489952    6000 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489955    6000 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489957    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489959    6000 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489962    6000 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489964    6000 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489966    6000 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489969    6000 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489971    6000 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489974    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489976    6000 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489978    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489981    6000 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489986    6000 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489989    6000 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 03 00:06:02.507656 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489991    6000 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489994    6000 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489996    6000 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.489999    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.490002    6000 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.490004    6000 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.490008    6000 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:02.490010    6000 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.490071    6000 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.491088    6000 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.491150    6000 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.491361    6000 server.go:1014] "Starting client certificate rotation"
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.491504    6000 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.491558    6000 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 23:59:50 +0000 UTC" deadline="2028-06-08 06:40:18.269257166 +0000 UTC"
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.491588    6000 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13998h34m15.777671379s"
Nov 03 00:06:02.508111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.491836    6000 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:06:02.508491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.492867    6000 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 03 00:06:02.508491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.494341    6000 log.go:25] "Validated CRI v1 runtime API"
Nov 03 00:06:02.508491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.496229    6000 log.go:25] "Validated CRI v1 image API"
Nov 03 00:06:02.508491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.496949    6000 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 03 00:06:02.508491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.499310    6000 fs.go:135] Filesystem UUIDs: map[0bfb0e16-cf90-4b9a-895e-e1dd3e36eb27:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 d0b66fa0-1bd3-4bcb-9fb5-ac2a56b63c80:/dev/nvme0n1p3]
Nov 03 00:06:02.508650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.499324    6000 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/05188748994e5aa0c19d7b7aa28777c3fe4f9e5ce6929e38759916fd66851a3d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/05188748994e5aa0c19d7b7aa28777c3fe4f9e5ce6929e38759916fd66851a3d/userdata/shm major:0 minor:365 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0834dacd29061203ceabf362cb927ce0ff242b9ba15067e37532738bf861b0a8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0834dacd29061203ceabf362cb927ce0ff242b9ba15067e37532738bf861b0a8/userdata/shm major:0 minor:373 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0b12439d2683c63cd82aaaab94c427b53e2049e0fa694de58d4b06d285c85921/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0b12439d2683c63cd82aaaab94c427b53e2049e0fa694de58d4b06d285c85921/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/14f8751721384b200f6703a22e3650e461d96900636af43d6326b88a0d5e3d1a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/14f8751721384b200f6703a22e3650e461d96900636af43d6326b88a0d5e3d1a/userdata/shm major:0 minor:430 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/27a4859046975425fd6db3d2a4a8be7ccd62dd628a0f6c53c738c0e7b935e0a0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/27a4859046975425fd6db3d2a4a8be7ccd62dd628a0f6c53c738c0e7b935e0a0/userdata/shm major:0 minor:499 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d558cc916bb56bf77c9928148473721927dae5a98a09e776641e31a271781ce/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d558cc916bb56bf77c9928148473721927dae5a98a09e776641e31a271781ce/userdata/shm major:0 minor:497 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3734545cae631fb2268cec7be98b800814ff2152cf2f238e079a66897fe9af88/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3734545cae631fb2268cec7be98b800814ff2152cf2f238e079a66897fe9af88/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/438f3006e3072384162919c0b3715eb3762d984664f2cf359f07dc4b50ea767f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/438f3006e3072384162919c0b3715eb3762d984664f2cf359f07dc4b50ea767f/userdata/shm major:0 minor:502 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/44f13221d6eab5752d88f5359f6df20161e29d0c61b4a859303519a6b1ccfb5a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/44f13221d6eab5752d88f5359f6df20161e29d0c61b4a859303519a6b1ccfb5a/userdata/shm major:0 minor:89 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4f29e92e99c391a6c65c933169ce7ac848971d22c52bcfc83318f74fb8117f67/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4f29e92e99c391a6c65c933169ce7ac848971d22c52bcfc83318f74fb8117f67/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74/userdata/shm major:0 minor:485 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/56b6fb778b1f8ef1b42a565ce08a5e0cc9bf01ac587771f5c41d878f7857d5d7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/56b6fb778b1f8ef1b42a565ce08a5e0cc9bf01ac587771f5c41d878f7857d5d7/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bcc239ed96c060607dda9a0190217573c63071fee14c4d87b4d079746759982/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bcc239ed96c060607dda9a0190217573c63071fee14c4d87b4d079746759982/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/77688afd59189f92aecc7e39fed0e42cff03efa97ea1cdc4a5f9d1eec716bb41/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/77688afd59189f92aecc7e39fed0e42cff03efa97ea1cdc4a5f9d1eec716bb41/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8b30957b0c0d2bf95621ac39ec85e881a2df7d721382ce5da02a2a78db6eb649/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8b30957b0c0d2bf95621ac39ec85e881a2df7d721382ce5da02a2a78db6eb649/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a0f0b7039e12be7087099855e4e23ed82a0d1a8e3e0236eff40ce66c6f4fabba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a0f0b7039e12be7087099855e4e23ed82a0d1a8e3e0236eff40ce66c6f4fabba/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9377decffd5c33c725c860453b1f61664e337161e17163c840c805ed4c06562/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9377decffd5c33c725c860453b1f61664e337161e17163c840c805ed4c06562/userdata/shm major:0 minor:294 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac7ef5ec05a7c587e3d3839a4f09664c26035bb9b2cf9e4c1e3412a9b0d01648/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac7ef5ec05a7c587e3d3839a4f09664c26035bb9b2cf9e4c1e3412a9b0d01648/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433/userdata/shm major:0 minor:325 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c45cec789f1f012c5bd82b86ecc1a9c5ecc18a047d585f47513efbd0935a40d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c45cec789f1f012c5bd82b86ecc1a9c5ecc18a047d585f47513efbd0935a40d3/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c61005ae1dade31ef22d219da9c2fcd052c154a2a4550a29470734a200332130/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c61005ae1dade31ef22d219da9c2fcd052c154a2a4550a29470734a200332130/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6a2b1bdef557c95ba59c06db8d52331ba8e867b9a9cd48a3a53c6c0dff7127e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6a2b1bdef557c95ba59c06db8d52331ba8e867b9a9cd48a3a53c6c0dff7127e/userdata/shm major:0 minor:292 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/df69d97fcb8d9ba5936c47cf0ad7c2be59314f358137217db2630f596eb99531/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/df69d97fcb8d9ba5936c47cf0ad7c2be59314f358137217db2630f596eb99531/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e3f6dbc10315280d5abb5d028f58c06fcdd9e4e77f8c79c447637eb742329a65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e3f6dbc10315280d5abb5d028f58c06fcdd9e4e77f8c79c447637eb742329a65/userdata/shm major:0 minor:405 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~projected/kube-api-access-vpp5m:{mountpoint:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~projected/kube-api-access-vpp5m major:0 minor:364 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:359 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:355 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:360 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:361 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:362 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:363 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04183b41-8425-43cd-a145-9e454673cad1/volumes/kubernetes.io~projected/kube-api-access-8xp29:{mountpoint:/var/lib/kubelet/pods/04183b41-8425-43cd-a145-9e454673cad1/volumes/kubernetes.io~projected/kube-api-access-8xp29 major:0 minor:498 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~projected/kube-api-access-v5z47:{mountpoint:/var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~projected/kube-api-access-v5z47 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~projected/kube-api-access-2w2j6:{mountpoint:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~projected/kube-api-access-2w2j6 major:0 minor:372 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:369 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:370 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:371 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volumes/kubernetes.io~projected/kube-api-access-9h6zb:{mountpoint:/var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volumes/kubernetes.io~projected/kube-api-access-9h6zb major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~projected/kube-api-access-bxrnc:{mountpoint:/var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~projected/kube-api-access-bxrnc major:0 minor:484 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:483 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:479 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/21ef2aeb-b54a-45f5-b58a-af731ebdfe11/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/21ef2aeb-b54a-45f5-b58a-af731ebdfe11/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~projected/kube-api-access-7wssr:{mountpoint:/var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~projected/kube-api-access-7wssr major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ce18949-1971-4017-ab1c-5faa6d7412b2/volumes/kubernetes.io~projected/kube-api-access-zxxbd:{mountpoint:/var/lib/kubelet/pods/3ce18949-1971-4017-ab1c-5faa6d7412b2/volumes/kubernetes.io~projected/kube-api-access-zxxbd major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ce18949-1971-4017-ab1c-5faa6d7412b2/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/3ce18949-1971-4017-ab1c-5faa6d7412b2/volumes/kubernetes.io~secret/metrics-certs major:0 minor:491 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6/volumes/kubernetes.io~projected/kube-api-access-lfgmh:{mountpoint:/var/lib/kubelet/pods/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6/volumes/kubernetes.io~projected/kube-api-access-lfgmh major:0 minor:378 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:375 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~projected/kube-api-access-7zv5n:{mountpoint:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~projected/kube-api-access-7zv5n major:0 minor:427 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/federate-client-tls:{mountpoint:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/federate-client-tls major:0 minor:420 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/secret-telemeter-client:{mountpoint:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/secret-telemeter-client major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/telemeter-client-tls:{mountpoint:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/telemeter-client-tls major:0 minor:426 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~projected/kube-api-access-rlf4r:{mountpoint:/var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~projected/kube-api-access-rlf4r major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~secret/prometheus-operator-tls:{mountpoint:/var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~secret/prometheus-operator-tls major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fedc973-716d-4cdc-9b47-748582eba143/volumes/kubernetes.io~projected/kube-api-access-vzz28:{mountpoint:/var/lib/kubelet/pods/5fedc973-716d-4cdc-9b47-748582eba143/volumes/kubernetes.io~projected/kube-api-access-vzz28 major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5fedc973-716d-4cdc-9b47-748582eba143/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/5fedc973-716d-4cdc-9b47-748582eba143/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f/volumes/kubernetes.io~projected/kube-api-access-nhx6r:{mountpoint:/var/lib/kubelet/pods/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f/volumes/kubernetes.io~projected/kube-api-access-nhx6r major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f/volumes/kubernetes.io~secret/cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7ad1f241-5476-4a43-a616-f26eff05c43d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/7ad1f241-5476-4a43-a616-f26eff05c43d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:496 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83c6f1f9-3e2f-4c6e-801f-29735ad355a2/volumes/kubernetes.io~projected/kube-api-access-9j7s6:{mountpoint:/var/lib/kubelet/pods/83c6f1f9-3e2f-4c6e-801f-29735ad355a2/volumes/kubernetes.io~projected/kube-api-access-9j7s6 major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c3c97a4-e610-4a34-9fe8-b5e754372cd2/volumes/kubernetes.io~projected/kube-api-access-ckjgg:{mountpoint:/var/lib/kubelet/pods/8c3c97a4-e610-4a34-9fe8-b5e754372cd2/volumes/kubernetes.io~projected/kube-api-access-ckjgg major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8c3c97a4-e610-4a34-9fe8-b5e754372cd2/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/8c3c97a4-e610-4a34-9fe8-b5e754372cd2/volumes/kubernetes.io~secret/metrics-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9df7b833-c25f-48d0-8de2-401b05bd6796/volumes/kubernetes.io~projected/kube-api-access-rgdjp:{mountpoint:/var/lib/kubelet/pods/9df7b833-c25f-48d0-8de2-401b05bd6796/volumes/kubernetes.io~projected/kube-api-access-rgdjp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~projected/kube-api-access-mjv8x:{mountpoint:/var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~projected/kube-api-access-mjv8x major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~projected/kube-api-access-zd8zx:{mountpoint:/var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~projected/kube-api-access-zd8zx major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b3876106-cc95-464e-9f2c-be6504f26585/volumes/kubernetes.io~projected/kube-api-access-hjr79:{mountpoint:/var/lib/kubelet/pods/b3876106-cc95-464e-9f2c-be6504f26585/volumes/kubernetes.io~projected/kube-api-access-hjr79 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0/volumes/kubernetes.io~projected/kube-api-access-zglcx:{mountpoint:/var/lib/kubelet/pods/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0/volumes/kubernetes.io~projected/kube-api-access-zglcx major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~projected/kube-api-access-n88np:{mountpoint:/var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~projected/kube-api-access-n88np major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2/volumes/kubernetes.io~projected/kube-api-access-s66w5:{mountpoint:/var/lib/kubelet/pods/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2/volumes/kubernetes.io~projected/kube-api-access-s66w5 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/alertmanager/9 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/config-reloader/6 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~empty-dir/config-out major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~projected/kube-api-access-l9xxv:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~projected/kube-api-access-l9xxv major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~projected/tls-assets major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/config-volume major:0 minor:320 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:319 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:321 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:318 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/web-config major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fba3da01-3b60-48ec-8d2d-2d90478e334a/volumes/kubernetes.io~projected/kube-api-access-l59sh:{mountpoint:/var/lib/kubelet/pods/fba3da01-3b60-48ec-8d2d-2d90478e334a/volumes/kubernetes.io~projected/kube-api-access-l59sh major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/f0d3378c94a4dd6e1f183eb9fa842d9cef734bbc179363449a3950c5ef73c13e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/50cf47a66189932e25761b2fa1ee9e34457916fec4a261d0311c8364a23948e6/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/6241799b888ccd42cb741aff6eea49e2a5b6de9e37c906bf9253cf48dbded22c/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/199eb88cb3df8b1e6e51f3de58227c51bbce25bccc2d5940337be0583f4b53de/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/693be9b5660ec3f07459dcce78102b01afe802c52756f47f76e32af8ffff0b15/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/ce1ce8dfc8b1ea4723b2798d3c66314b34529f2320a2e7bcca560c31d874c315/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/fb20204bedf28179f3953ca5ff04b268a332f5b1e477698d7d62621ec955af2d/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c26a766b452e24abc7e9acf0400934d89efd6b88e0f14fb505619dd4eac15050/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/d47ac40e05686d46e9cd511883538c74811c3fafe77bf09cbc4c4e992e1abd0a/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/5304331f63977aac71cfeb2dd2201c75866733d4bf8a4a0cbd7687bf4459e78b/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5517ec52fba374db7128f2ecfc9aa7120dc381f5f14783c1ef0290674f3fb002/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/4e06df0bb2b45c1c33edb3c8204d063248cda520b5d3b920d736204a048f223c/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-149:{mountpoint:/var/lib/containers/storage/overlay/2d65d0a8616a4acd5a8dd67a75ca92d23114a2904c7b9c4a6fbcbf5fb887e992/merged major:0 minor:149 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/ece646c2828efd161d3a2692770a496212692336164c6a285577547c9224631c/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/661113add22c842dbc5975ea58c3c08449d69e68ab6166fa509ee8e827284b24/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/cd8da5463c7802cdc7f001aea91f395f712e52af54a3919ffa0973eefb16a015/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/67ff54e62dd878c4421a8ebca03f6ffc4d3982ea1c3453090ea28f6c2e914173/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/77b4251af3b77a8f3d1e32caecde35fba52cfec31bc2dc6ae50736a0d121be82/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/d73aaa764e12f8230f4fb673e8ebc1661cd0f6f7e47babd7339eceb39fcd5cbc/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/046fdd22e72a8a2e4d12a493247db4e777a2ad22e9c8b06280b8b89ee4d4ccdc/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f42fcc88dc5de813a34a09cabec47c867bd51b2e1b0be403f19b7d3589f29851/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/761edaee1d509ba95cda861108d87a660854aa377e033975813d39a533a5abff/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/eef1b8f0fb0080ba9b204ffa7686a13be7dbf71085e4cfea7bdb3b2bff3f6370/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/b85618d60827906418b97bbf2725471cbf15dac55482ec3e5092a1f459d292ca/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/70a1322bac47065a7e6e8e5b0b70f0d5b0af305c8a9b828f9560481fb43c82e3/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/2b4b1da41448a7743d45818c03485bed1632c1513edba9430b27beee4b1cc9d7/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/bfaf47ae70541181b539c0e94657a674f6affc9105fa0c759b327dcf57b212c1/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/1a8167515af87300675938288192a3f4ff0cba4b25edaeafe7331611ed1a7dc4/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/30b53230064a703e1a6e8474a98f28140b145aba2daeca43576b68490634a9dd/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/be72509f62d36444d3de4d35b5aabdb05bb53c18fc3d84b4743a2f35b98f0843/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-253:{mountpoint:/var/lib/containers/storage/overlay/0bc1b77d920f940ee608d0ec3d7e12d15b37c805d01e195ce874ba975104ae25/merged major:0 minor:253 fsType:overlay blockSize:0} overlay_0-255:{mountpoint:/var/lib/containers/storage/overlay/3672a6798be2cbc00d89c352994baa7a26b98290cfdcaa7b05d9e8d3a18e4b1c/merged major:0 minor:255 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/b42bfaa148bd71d8fd7d47ea04c0fc0de7fd43ea237116b3f899f70b0586eaab/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/de05d8671a218093e402aa958b67dbc4a76b504113f8bb9b0b9ea613b2a4e043/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/459e34f8550f6256c6aa321f8d918c2524e2a7676e0aa06a60951c60e857a053/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/a8c2fc6107c4ba60a68e3c252f4ae0b30fd0088d5a34c6af48673e87a9210f5e/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/78b555b88f537f6eff8adf853fb39074bcd72366752c0bf83f3a43ab822b3fa8/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/4d2a9a96561c6a7eb35d50fd3838d9fa6a853986b3b6fa2e7e12ca3330bd186e/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/d5c9df2a10b8550a152713e0fae1b5f8102f81c2740e0a5d6c96ba93041e0ab7/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/69202f3ee4734edd9a40b17fab1c7f0d749779ed85610d992c72462455688a04/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-323:{mountpoint:/var/lib/containers/storage/overlay/482c700ed58f6a2ad992a150214f7e8d72f93dc119dfb66442b6f64ea28fef3e/merged major:0 minor:323 fsType:overlay blockSize:0} overlay_0-327:{mountpoint:/var/lib/containers/storage/overlay/267c7b96ba19eb04c0856125768b1f85f843d3b9aaa5f9e1fc91d2b80947e43b/merged major:0 minor:327 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/052cfc949e9c2991308e014b42f527cf3b53cc3e30c85bf4c3fc7a59b3538e62/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-338:{mountpoint:/var/lib/containers/storage/overlay/7a12c046d04cc0adb28b4c5b753eb03fc68721aa753ff1f60b394c71c6b57261/merged major:0 minor:338 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/5455872bbfa2f8c6cf6347cccafe2c49d2089e3f5e5af29e09da4e2328d65359/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/a4769ad6a0c9b60b716a57ab09332742746d57c2793bcaf8a93a43e73fa00a41/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/d30fc464b26066087222411218addde93be68c3c4787422045318d2bcceca0d2/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/8c202df3e3f1afaf886dd5aa4d136b25c00f48475db3f3ab5477df1ac2a41a68/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/b419f943df57e9b2eba4b1287b01c20af7c7f7ec2a843602a3be9c58565786b5/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-379:{mountpoint:/var/lib/containers/storage/overlay/7e127dbfed23f574799489582a1c53ec43ea2eecdb16c19c50e0510c9f87fdbc/merged major:0 minor:379 fsType:overlay blockSize:0} overlay_0-381:{mountpoint:/var/lib/containers/storage/overlay/27a8b78cb09f963a69635991aea64302cdf15dc8b0f07d4bd2179eaecbdd330d/merged major:0 minor:381 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/276eadbd1f6cd6391e693f42e7613fed4a2a724cd78c47572489e04c8794bc97/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-393:{mountpoint:/var/lib/containers/storage/overlay/c2c892fe73bd9bc8f662cdc9b23ef1c601a46eb3ec907e416288537ecbfdac94/merged major:0 minor:393 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/a9da4191a5c4563fef7637d1afa882bd412cac170bd64e15c58109ccaac28790/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-403:{mountpoint:/var/lib/containers/storage/overlay/28b6900a71976236139a1f4e81563b5fd40626bab60c1c75f3e431563a2e4430/merged major:0 minor:403 fsType:overlay blockSize:0} overlay_0-407:{mountpoint:/var/lib/containers/storage/overlay/92330d3f8954abde0e417a0821453dc688db20c12975bd6ebf5afe5db7a150e8/merged major:0 minor:407 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/4ef2ebc3569a2b2931323354adb45ff9f55e88be0f61126d30b86e87bda40694/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-428:{mountpoint:/var/lib/containers/storage/overlay/4ad920d33ac9199d118a5c6ff98a3877dcb6720ad7c8ccec46a081a371776575/merged major:0 minor:428 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/7d52937d62332be8686a8c775469a5a4a53b568c58706c748b229fdffd25c7ee/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/d78f83312eb4ccb02923c1c223cf5476eab8603a47c5cbb656dbe3cbe679d255/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/2562a7da2f6a7813056805e10aceb07495a62dde1fca2c650291f546b5f56128/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/f17536396d0fd78bf270967c3d8c472b69ab885a76ad25fe8d7d2dd97eb0e529/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/1d17c2522868774229dae36a3bc76b05058b4da4e10fd8031c466c003adc96ab/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/578fa25f918053ee6bfb3cf4f8a29464ae8f2befab4c2633ec5d86e285836215/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-456:{mountpoint:/var/lib/containers/storage/overlay/4940f17fb2b0d935fc0e3a7e078d27ea560dbc3d568e31e196b9cfb62b9755b4/merged major:0 minor:456 fsType:overlay blockSize:0} overlay_0-458:{mountpoint:/var/lib/containers/storage/overlay/9c57cdd12cb739403f7d9c2d7bc74fc9d2c7772ef2a7b33ab862a82453ad3250/merged major:0 minor:458 fsType:overlay blockSize:0} overlay_0-468:{mountpoint:/var/lib/containers/storage/overlay/e11b396dd1bcbf5dc8ddff29f1384dd6d77291376d94a45c03bbeaa6a06dab20/merged major:0 minor:468 fsType:overlay blockSize:0} overlay_0-474:{mountpoint:/var/lib/containers/storage/overlay/bb5123e199961db5932cd93c2c4b601d978409c9305ca7352dfa9d4487da1dd9/merged major:0 minor:474 fsType:overlay blockSize:0} overlay_0-487:{mountpoint:/var/lib/containers/storage/overlay/6deb4a3a15fc9d76f900f221027e787d5e0e40c212f3306aecc425f7567f730c/merged major:0 minor:487 fsType:overlay blockSize:0} overlay_0-489:{mountpoint:/var/lib/containers/storage/overlay/929a887db2c972c69f0e53b17c51b511504eade990a25bea1c72d8467b55352a/merged major:0 minor:489 fsType:overlay blockSize:0} overlay_0-503:{mountpoint:/var/lib/containers/storage/overlay/c1e6879e84d5f09563d05c3ba3e50ab35af332ed5be7ce80ab0f51d7096d4972/merged major:0 minor:503 fsType:overlay blockSize:0} overlay_0-506:{mountpoint:/var/lib/containers/storage/overlay/7cb9d612f92672fd9cf120459282eacfb17ddafd148d23fbb62afdfa19a9f90c/merged major:0 minor:506 fsType:overlay blockSize:0} overlay_0-508:{mountpoint:/var/lib/containers/storage/overlay/c84de87dfa6cd598170fc149d919378399e1d833d5ae9c6f3ea5301e9ccb1f49/merged major:0 minor:508 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9e1d1f9b154dbcb21ae90122086c37595bc3c2a83fdd87a9f094bfba4ee90c33/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-510:{mountpoint:/var/lib/containers/storage/overlay/663080947d0ac3d802b84ee095be5808121ad575df9663763c84ccc5c3511c09/merged major:0 minor:510 fsType:overlay blockSize:0} overlay_0-512:{mountpoint:/var/lib/containers/storage/overlay/05961371ef06cc0f8a1af88018b55cc3cba1f7680eb12d6bd6fdf16d7c7b30bc/merged major:0 minor:512 fsType:overlay blockSize:0} overlay_0-518:{mountpoint:/var/lib/containers/storage/overlay/ba5f1a06b13401e46d0f721ae6a417e0faa5430d4b63fa8f14cea82f1eddc2d3/merged major:0 minor:518 fsType:overlay blockSize:0} overlay_0-523:{mountpoint:/var/lib/containers/storage/overlay/c12d74ae712b45a8835641052854d0ebb2a1a7219e80d644370eed1c71f542e5/merged major:0 minor:523 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/95fabfcec1a205f440a99b5cb7e8f885882ebbbca1ff907b8917f61a4a9741ae/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/b762602a6a689b50bd82e3ee46208aefc3ec6ba925dd10136959f88295893be7/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/8f334b5baff7a2d281631a7f0cda9eaf64802b1258c5d889b04f99a76ad7a046/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/6eed0c43a5cfc60a859da91a278b63ec130b52cb67028b100eec987f5b4dbc18/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/6504853492e938fc6a98d9d8bf20d202c69a197f270ac0740349aa93f7b27f85/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/a89ba191b1470cd23ddbde44735a1a3a83ff1941576ac7885d9e2d5957fa4836/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/951d9f36fb9e8a66ab255b53077a9296fa7cf8e5d9fcd7293d98e896904a1b74/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/d244201b77bd07b3bb7fde0498473f71787c1c0c53ecaecd646ee4636e2ef742/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/c73ecb43faedef63cd6dace9ad54bb4773ba4b126f3947c7503afdbefe6f3c51/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 03 00:06:02.509526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.509045    6000 manager.go:217] Machine: {Timestamp:2026-11-03 00:06:02.508582299 +0000 UTC m=+0.063934313 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3107624 MemoryCapacity:32812011520 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec274b6b8202b92f9a120b00f0fb0c12 SystemUUID:ec274b6b-8202-b92f-9a12-0b00f0fb0c12 BootID:0da4d909-5ff0-4ea5-b942-2a8140f2c6c7 Filesystems:[{Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c61005ae1dade31ef22d219da9c2fcd052c154a2a4550a29470734a200332130/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:285 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~projected/kube-api-access-7zv5n DeviceMajor:0 DeviceMinor:427 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/438f3006e3072384162919c0b3715eb3762d984664f2cf359f07dc4b50ea767f/userdata/shm DeviceMajor:0 DeviceMinor:502 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/5fedc973-716d-4cdc-9b47-748582eba143/volumes/kubernetes.io~projected/kube-api-access-vzz28 DeviceMajor:0 DeviceMinor:204 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8b30957b0c0d2bf95621ac39ec85e881a2df7d721382ce5da02a2a78db6eb649/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:369 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-407 DeviceMajor:0 DeviceMinor:407 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a0f0b7039e12be7087099855e4e23ed82a0d1a8e3e0236eff40ce66c6f4fabba/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-253 DeviceMajor:0 DeviceMinor:253 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:316 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/04183b41-8425-43cd-a145-9e454673cad1/volumes/kubernetes.io~projected/kube-api-access-8xp29 DeviceMajor:0 DeviceMinor:498 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9377decffd5c33c725c860453b1f61664e337161e17163c840c805ed4c06562/userdata/shm DeviceMajor:0 DeviceMinor:294 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~projected/kube-api-access-7wssr DeviceMajor:0 DeviceMinor:246 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-379 DeviceMajor:0 DeviceMinor:379 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:479 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:310 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-468 DeviceMajor:0 DeviceMinor:468 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/44f13221d6eab5752d88f5359f6df20161e29d0c61b4a859303519a6b1ccfb5a/userdata/shm DeviceMajor:0 DeviceMinor:89 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~secret/prometheus-operator-tls DeviceMajor:0 DeviceMinor:225 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-403 DeviceMajor:0 DeviceMinor:403 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/federate-client-tls DeviceMajor:0 DeviceMinor:420 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/secret-telemeter-client DeviceMajor:0 DeviceMinor:425 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ce18949-1971-4017-ab1c-5faa6d7412b2/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:491 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:286 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~projected/kube-api-access-bxrnc DeviceMajor:0 DeviceMinor:484 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74/userdata/shm DeviceMajor:0 DeviceMinor:485 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-503 DeviceMajor:0 DeviceMinor:503 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:319 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-428 DeviceMajor:0 DeviceMinor:428 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~projected/kube-api-access-2w2j6 DeviceMajor:0 DeviceMinor:372 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-506 DeviceMajor:0 DeviceMinor:506 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3ce18949-1971-4017-ab1c-5faa6d7412b2/volumes/kubernetes.io~projected/kube-api-access-zxxbd DeviceMajor:0 DeviceMinor:60 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bcc239ed96c060607dda9a0190217573c63071fee14c4d87b4d079746759982/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~projected/kube-api-access-v5z47 DeviceMajor:0 DeviceMinor:289 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:360 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:315 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:483 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/8c3c97a4-e610-4a34-9fe8-b5e754372cd2/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0834dacd29061203ceabf362cb927ce0ff242b9ba15067e37532738bf861b0a8/userdata/shm DeviceMajor:0 DeviceMinor:373 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-381 DeviceMajor:0 DeviceMinor:381 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:317 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-338 DeviceMajor:0 DeviceMinor:338 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/14f8751721384b200f6703a22e3650e461d96900636af43d6326b88a0d5e3d1a/userdata/shm DeviceMajor:0 DeviceMinor:430 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volumes/kubernetes.io~projected/kube-api-access-9h6zb DeviceMajor:0 DeviceMinor:65 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/83c6f1f9-3e2f-4c6e-801f-29735ad355a2/volumes/kubernetes.io~projected/kube-api-access-9j7s6 DeviceMajor:0 DeviceMinor:67 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:315 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:318 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/9df7b833-c25f-48d0-8de2-401b05bd6796/volumes/kubernetes.io~projected/kube-api-access-rgdjp DeviceMajor:0 DeviceMinor:59 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0b12439d2683c63cd82aaaab94c427b53e2049e0fa694de58d4b06d285c85921/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b3876106-cc95-464e-9f2c-be6504f26585/volumes/kubernetes.io~projected/kube-api-access-hjr79 DeviceMajor:0 DeviceMinor:63 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:245 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~projected/kube-api-access-l9xxv DeviceMajor:0 DeviceMinor:322 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:359 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~projected/kube-api-access-n88np DeviceMajor:0 DeviceMinor:288 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~projected/kube-api-access-vpp5m DeviceMajor:0 DeviceMinor:364 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:315 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/secret-telemeter-client-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:424 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fba3da01-3b60-48ec-8d2d-2d90478e334a/volumes/kubernetes.io~projected/kube-api-access-l59sh DeviceMajor:0 DeviceMinor:66 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/df69d97fcb8d9ba5936c47cf0ad7c2be59314f358137217db2630f596eb99531/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:362 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-458 DeviceMajor:0 DeviceMinor:458 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-487 DeviceMajor:0 DeviceMinor:487 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac7ef5ec05a7c587e3d3839a4f09664c26035bb9b2cf9e4c1e3412a9b0d01648/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-149 DeviceMajor:0 DeviceMinor:149 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8c3c97a4-e610-4a34-9fe8-b5e754372cd2/volumes/kubernetes.io~projected/kube-api-access-ckjgg DeviceMajor:0 DeviceMinor:197 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~secret/prometheus-operator-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:229 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:363 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6/volumes/kubernetes.io~projected/kube-api-access-lfgmh DeviceMajor:0 DeviceMinor:378 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-255 DeviceMajor:0 DeviceMinor:255 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0da10585-07cb-473c-94c2-d9a8c519ea73/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:277 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/dcd7e2a0-6e98-4417-bbc9-25868b1ca911/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:282 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/1bf749d7-4d5c-4855-b945-fbf6b4953624/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c45cec789f1f012c5bd82b86ecc1a9c5ecc18a047d585f47513efbd0935a40d3/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3734545cae631fb2268cec7be98b800814ff2152cf2f238e079a66897fe9af88/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:355 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/03ed77f9-5e31-428a-988c-7811b60e299a/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:361 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-393 DeviceMajor:0 DeviceMinor:393 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-510 DeviceMajor:0 DeviceMinor:510 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0/volumes/kubernetes.io~projected/kube-api-access-zglcx DeviceMajor:0 DeviceMinor:64 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4f29e92e99c391a6c65c933169ce7ac848971d22c52bcfc83318f74fb8117f67/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/21ef2aeb-b54a-45f5-b58a-af731ebdfe11/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5a1b5632-bc8c-4692-8e60-816ef5b35575/volumes/kubernetes.io~projected/kube-api-access-rlf4r DeviceMajor:0 DeviceMinor:232 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:321 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:375 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e3f6dbc10315280d5abb5d028f58c06fcdd9e4e77f8c79c447637eb742329a65/userdata/shm DeviceMajor:0 DeviceMinor:405 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-456 DeviceMajor:0 DeviceMinor:456 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-508 DeviceMajor:0 DeviceMinor:508 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/56b6fb778b1f8ef1b42a565ce08a5e0cc9bf01ac587771f5c41d878f7857d5d7/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:241 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-323 DeviceMajor:0 DeviceMinor:323 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:315 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/50e89fdd-eb0a-44c4-b904-9e7955f88504/volumes/kubernetes.io~secret/telemeter-client-tls DeviceMajor:0 DeviceMinor:426 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:202 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~projected/kube-api-access-mjv8x DeviceMajor:0 DeviceMinor:287 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:370 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/182c2fcb-792e-429f-9ab3-875803a3fd66/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:371 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/7ad1f241-5476-4a43-a616-f26eff05c43d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:496 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-523 DeviceMajor:0 DeviceMinor:523 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2/volumes/kubernetes.io~projected/kube-api-access-s66w5 DeviceMajor:0 DeviceMinor:61 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6a2b1bdef557c95ba59c06db8d52331ba8e867b9a9cd48a3a53c6c0dff7127e/userdata/shm DeviceMajor:0 DeviceMinor:292 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:314 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433/userdata/shm DeviceMajor:0 DeviceMinor:325 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-518 DeviceMajor:0 DeviceMinor:518 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5fedc973-716d-4cdc-9b47-748582eba143/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f/volumes/kubernetes.io~projected/kube-api-access-nhx6r DeviceMajor:0 DeviceMinor:203 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/05188748994e5aa0c19d7b7aa28777c3fe4f9e5ce6929e38759916fd66851a3d/userdata/shm DeviceMajor:0 DeviceMinor:365 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-474 DeviceMajor:0 DeviceMinor:474 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-489 DeviceMajor:0 DeviceMinor:489 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/77688afd59189f92aecc7e39fed0e42cff03efa97ea1cdc4a5f9d1eec716bb41/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:284 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-512 DeviceMajor:0 DeviceMinor:512 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:320 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b24caade-1c7b-44e6-b04e-fbc0c44403fe/volumes/kubernetes.io~projected/kube-api-access-zd8zx DeviceMajor:0 DeviceMinor:62 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:310 Capacity:31633412096 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-327 DeviceMajor:0 DeviceMinor:327 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d558cc916bb56bf77c9928148473721927dae5a98a09e776641e31a271781ce/userdata/shm DeviceMajor:0 DeviceMinor:497 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/27a4859046975425fd6db3d2a4a8be7ccd62dd628a0f6c53c738c0e7b935e0a0/userdata/shm DeviceMajor:0 DeviceMinor:499 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:05188748994e5aa MacAddress:3e:cb:e4:34:a7:e0 Speed:10000 Mtu:8901} {Name:0834dacd2906120 MacAddress:2e:d4:b7:07:a1:21 Speed:10000 Mtu:8901} {Name:14f8751721384b2 MacAddress:22:dc:13:26:a0:eb Speed:10000 Mtu:8901} {Name:27a485904697542 MacAddress:c2:90:5a:54:e8:5e Speed:10000 Mtu:8901} {Name:2d558cc916bb56b MacAddress:fe:0b:8c:74:db:b6 Speed:10000 Mtu:8901} {Name:3734545cae631fb MacAddress:da:31:7e:fb:33:63 Speed:10000 Mtu:8901} {Name:438f3006e307238 MacAddress:0e:62:5f:2a:86:68 Speed:10000 Mtu:8901} {Name:46a2bef15f0cdac MacAddress:0e:10:6e:92:85:7b Speed:10000 Mtu:8901} {Name:513f34b7127724c MacAddress:ea:7d:8d:65:11:93 Speed:10000 Mtu:8901} {Name:5bcc239ed96c060 MacAddress:8e:53:ca:e5:7d:7c Speed:10000 Mtu:8901} {Name:8b30957b0c0d2bf MacAddress:52:e1:ee:84:dd:eb Speed:10000 Mtu:8901} {Name:bcc200aad21f605 MacAddress:d2:eb:ff:7d:33:7a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e1:8a:73:4d Speed:0 Mtu:9001} {Name:br-int MacAddress:92:2a:3e:21:66:4c Speed:0 Mtu:8901} {Name:c61005ae1dade31 MacAddress:52:b4:c2:81:8f:c5 Speed:10000 Mtu:8901} {Name:c6a2b1bdef557c9 MacAddress:d6:1a:51:8d:7d:56 Speed:10000 Mtu:8901} {Name:df69d97fcb8d9ba MacAddress:6e:d6:06:e5:98:8b Speed:10000 Mtu:8901} {Name:e3f6dbc10315280 MacAddress:46:03:dc:19:c3:d6 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e1:8a:73:4d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:1e:37:ae:30:c8:a5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:de:e9:75:90:61:41 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812011520 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 03 00:06:02.509763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.509532    6000 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 03 00:06:02.509763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.509581    6000 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 03 00:06:02.509883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.509859    6000 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 03 00:06:02.510015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.509884    6000 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-136-97.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 03 00:06:02.510056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510017    6000 topology_manager.go:143] "Creating topology manager with none policy"
Nov 03 00:06:02.510056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510026    6000 container_manager_linux.go:311] "Creating device plugin manager"
Nov 03 00:06:02.510056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510035    6000 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:06:02.510056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510056    6000 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 03 00:06:02.510191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510151    6000 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 03 00:06:02.510335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510326    6000 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 03 00:06:02.510450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510442    6000 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 03 00:06:02.510484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510478    6000 kubelet.go:493] "Attempting to sync node with API server"
Nov 03 00:06:02.510514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510491    6000 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 03 00:06:02.510514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510501    6000 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 03 00:06:02.510514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510510    6000 kubelet.go:405] "Adding apiserver pod source"
Nov 03 00:06:02.510597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.510527    6000 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 03 00:06:02.511227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.511214    6000 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:06:02.511276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.511228    6000 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:06:02.511720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.511696    6000 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 03 00:06:02.514329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514312    6000 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 03 00:06:02.514329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514326    6000 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 03 00:06:02.514480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514438    6000 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 03 00:06:02.514670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514655    6000 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 03 00:06:02.514732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514695    6000 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 03 00:06:02.514820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514809    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 03 00:06:02.514873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514829    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 03 00:06:02.514873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514839    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 03 00:06:02.514873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514849    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 03 00:06:02.514873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514859    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 03 00:06:02.514873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514870    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514889    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514897    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514909    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514918    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514941    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514956    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.514997    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 03 00:06:02.515152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.515007    6000 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 03 00:06:02.518485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.518470    6000 server.go:1263] "Started kubelet"
Nov 03 00:06:02.518601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.518538    6000 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 03 00:06:02.518649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.518568    6000 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 03 00:06:02.518697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.518650    6000 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 03 00:06:02.519001 ip-10-0-136-97 systemd[1]: Started Kubernetes Kubelet.
Nov 03 00:06:02.522718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.519533    6000 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 03 00:06:02.522718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.521171    6000 server.go:317] "Adding debug handlers to kubelet server"
Nov 03 00:06:02.524188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.524170    6000 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 03 00:06:02.524299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.524202    6000 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 03 00:06:02.524377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.524345    6000 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 03 00:06:02.524377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.524359    6000 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 03 00:06:02.524473 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:02.524406    6000 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-136-97.ec2.internal\" not found"
Nov 03 00:06:02.524520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.524496    6000 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 03 00:06:02.524776 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.524598    6000 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 23:59:52 +0000 UTC" deadline="2028-04-19 02:24:41.923782864 +0000 UTC"
Nov 03 00:06:02.524776 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.524623    6000 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12794h18m39.399163289s"
Nov 03 00:06:02.527232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527212    6000 factory.go:55] Registering systemd factory
Nov 03 00:06:02.527232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527234    6000 factory.go:223] Registration of the systemd container factory successfully
Nov 03 00:06:02.527535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527516    6000 factory.go:153] Registering CRI-O factory
Nov 03 00:06:02.527535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527535    6000 factory.go:223] Registration of the crio container factory successfully
Nov 03 00:06:02.527687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527597    6000 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 03 00:06:02.527687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527616    6000 factory.go:103] Registering Raw factory
Nov 03 00:06:02.527687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527627    6000 manager.go:1199] Started watching for new ooms in manager
Nov 03 00:06:02.527840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527806    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" volumeName="kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:06:02.527895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527846    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 00:06:02.527895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527860    6000 manager.go:319] Starting recovery of all containers
Nov 03 00:06:02.527895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527861    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext=""
Nov 03 00:06:02.527895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527876    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 00:06:02.527895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527889    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bf749d7-4d5c-4855-b945-fbf6b4953624" volumeName="kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-config" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527901    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ce8d43-d4b6-4b23-a235-624a0e30612d" volumeName="kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527913    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ce8d43-d4b6-4b23-a235-624a0e30612d" volumeName="kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527927    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6" volumeName="kubernetes.io/projected/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-kube-api-access-lfgmh" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527940    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527952    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-tls" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527964    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182c2fcb-792e-429f-9ab3-875803a3fd66" volumeName="kubernetes.io/projected/182c2fcb-792e-429f-9ab3-875803a3fd66-kube-api-access-2w2j6" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527975    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.527988    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f" volumeName="kubernetes.io/secret/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-cert" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528000    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0" volumeName="kubernetes.io/empty-dir/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-tmp-dir" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528016    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528033    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528045    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/configmap/03ed77f9-5e31-428a-988c-7811b60e299a-metrics-client-ca" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528072    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da10585-07cb-473c-94c2-d9a8c519ea73" volumeName="kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-tls" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528084    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da10585-07cb-473c-94c2-d9a8c519ea73" volumeName="kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528098    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182c2fcb-792e-429f-9ab3-875803a3fd66" volumeName="kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-metrics-server-audit-profiles" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528110    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182c2fcb-792e-429f-9ab3-875803a3fd66" volumeName="kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-tls" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528123    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ef2aeb-b54a-45f5-b58a-af731ebdfe11" volumeName="kubernetes.io/configmap/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-konnectivity-ca" seLinuxMountContext=""
Nov 03 00:06:02.528130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528135    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="21ef2aeb-b54a-45f5-b58a-af731ebdfe11" volumeName="kubernetes.io/secret/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-agent-certs" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528146    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6" volumeName="kubernetes.io/secret/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-monitoring-plugin-cert" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528158    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da10585-07cb-473c-94c2-d9a8c519ea73" volumeName="kubernetes.io/empty-dir/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-textfile" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528171    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528183    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528195    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528215    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528227    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ce8d43-d4b6-4b23-a235-624a0e30612d" volumeName="kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528238    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1b5632-bc8c-4692-8e60-816ef5b35575" volumeName="kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528268    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1b5632-bc8c-4692-8e60-816ef5b35575" volumeName="kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-tls" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528285    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1b5632-bc8c-4692-8e60-816ef5b35575" volumeName="kubernetes.io/configmap/5a1b5632-bc8c-4692-8e60-816ef5b35575-metrics-client-ca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528297    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fedc973-716d-4cdc-9b47-748582eba143" volumeName="kubernetes.io/secret/5fedc973-716d-4cdc-9b47-748582eba143-insights-runtime-extractor-tls" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528307    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c3c97a4-e610-4a34-9fe8-b5e754372cd2" volumeName="kubernetes.io/projected/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-kube-api-access-ckjgg" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528319    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9df7b833-c25f-48d0-8de2-401b05bd6796" volumeName="kubernetes.io/projected/9df7b833-c25f-48d0-8de2-401b05bd6796-kube-api-access-rgdjp" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528330    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2de66e0-f1f5-44d4-8eca-3bd623d6fd84" volumeName="kubernetes.io/projected/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-kube-api-access-mjv8x" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528343    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24caade-1c7b-44e6-b04e-fbc0c44403fe" volumeName="kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-tuned" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528353    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528365    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba3da01-3b60-48ec-8d2d-2d90478e334a" volumeName="kubernetes.io/projected/fba3da01-3b60-48ec-8d2d-2d90478e334a-kube-api-access-l59sh" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528377    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04183b41-8425-43cd-a145-9e454673cad1" volumeName="kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528389    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bf749d7-4d5c-4855-b945-fbf6b4953624" volumeName="kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-script-lib" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528401    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528413    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528426    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-serving-certs-ca-bundle" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528437    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fedc973-716d-4cdc-9b47-748582eba143" volumeName="kubernetes.io/projected/5fedc973-716d-4cdc-9b47-748582eba143-kube-api-access-vzz28" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528449    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83c6f1f9-3e2f-4c6e-801f-29735ad355a2" volumeName="kubernetes.io/configmap/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-iptables-alerter-script" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528461    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2de66e0-f1f5-44d4-8eca-3bd623d6fd84" volumeName="kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-tls" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528472    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bf749d7-4d5c-4855-b945-fbf6b4953624" volumeName="kubernetes.io/projected/1bf749d7-4d5c-4855-b945-fbf6b4953624-kube-api-access-9h6zb" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528485    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528498    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ce8d43-d4b6-4b23-a235-624a0e30612d" volumeName="kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528509    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c3c97a4-e610-4a34-9fe8-b5e754372cd2" volumeName="kubernetes.io/configmap/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-config-volume" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528521    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcd7e2a0-6e98-4417-bbc9-25868b1ca911" volumeName="kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-metrics-client-ca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528533    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcd7e2a0-6e98-4417-bbc9-25868b1ca911" volumeName="kubernetes.io/empty-dir/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-volume-directive-shadow" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528545    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" volumeName="kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528558    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528580    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bf749d7-4d5c-4855-b945-fbf6b4953624" volumeName="kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-env-overrides" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528592    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f" volumeName="kubernetes.io/projected/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-kube-api-access-nhx6r" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528604    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3876106-cc95-464e-9f2c-be6504f26585" volumeName="kubernetes.io/projected/b3876106-cc95-464e-9f2c-be6504f26585-kube-api-access-hjr79" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528614    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528627    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528640    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/empty-dir/03ed77f9-5e31-428a-988c-7811b60e299a-tmp" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528653    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da10585-07cb-473c-94c2-d9a8c519ea73" volumeName="kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-metrics-client-ca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528666    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1d1799f5-bac5-4c4c-b399-6f1a5130e140" volumeName="kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528678    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-federate-client-tls" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528690    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c3c97a4-e610-4a34-9fe8-b5e754372cd2" volumeName="kubernetes.io/empty-dir/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-tmp-dir" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528701    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24caade-1c7b-44e6-b04e-fbc0c44403fe" volumeName="kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-tmp" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528715    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3876106-cc95-464e-9f2c-be6504f26585" volumeName="kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-multus-daemon-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528729    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcd7e2a0-6e98-4417-bbc9-25868b1ca911" volumeName="kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528741    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce18949-1971-4017-ab1c-5faa6d7412b2" volumeName="kubernetes.io/projected/3ce18949-1971-4017-ab1c-5faa6d7412b2-kube-api-access-zxxbd" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528752    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-metrics-client-ca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528764    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528775    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fedc973-716d-4cdc-9b47-748582eba143" volumeName="kubernetes.io/configmap/5fedc973-716d-4cdc-9b47-748582eba143-kube-rbac-proxy-cm" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528788    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" volumeName="kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528799    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528824    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528837    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fba3da01-3b60-48ec-8d2d-2d90478e334a" volumeName="kubernetes.io/configmap/fba3da01-3b60-48ec-8d2d-2d90478e334a-serviceca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528850    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/projected/03ed77f9-5e31-428a-988c-7811b60e299a-kube-api-access-vpp5m" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528862    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da10585-07cb-473c-94c2-d9a8c519ea73" volumeName="kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-accelerators-collector-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528877    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182c2fcb-792e-429f-9ab3-875803a3fd66" volumeName="kubernetes.io/empty-dir/182c2fcb-792e-429f-9ab3-875803a3fd66-audit-log" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528888    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ce8d43-d4b6-4b23-a235-624a0e30612d" volumeName="kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528900    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528913    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528926    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0da10585-07cb-473c-94c2-d9a8c519ea73" volumeName="kubernetes.io/projected/0da10585-07cb-473c-94c2-d9a8c519ea73-kube-api-access-v5z47" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528940    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ce8d43-d4b6-4b23-a235-624a0e30612d" volumeName="kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528953    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-client-tls" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528965    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8c3c97a4-e610-4a34-9fe8-b5e754372cd2" volumeName="kubernetes.io/secret/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-metrics-tls" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528976    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2de66e0-f1f5-44d4-8eca-3bd623d6fd84" volumeName="kubernetes.io/configmap/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-metrics-client-ca" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528988    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0" volumeName="kubernetes.io/projected/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-kube-api-access-zglcx" seLinuxMountContext=""
Nov 03 00:06:02.528964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.528999    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcd7e2a0-6e98-4417-bbc9-25868b1ca911" volumeName="kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529011    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529025    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32ce8d43-d4b6-4b23-a235-624a0e30612d" volumeName="kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529038    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529049    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83c6f1f9-3e2f-4c6e-801f-29735ad355a2" volumeName="kubernetes.io/projected/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-kube-api-access-9j7s6" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529061    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2de66e0-f1f5-44d4-8eca-3bd623d6fd84" volumeName="kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529073    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcd7e2a0-6e98-4417-bbc9-25868b1ca911" volumeName="kubernetes.io/projected/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-api-access-n88np" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529085    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dcd7e2a0-6e98-4417-bbc9-25868b1ca911" volumeName="kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-tls" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529097    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" volumeName="kubernetes.io/projected/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-kube-api-access-s66w5" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529109    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182c2fcb-792e-429f-9ab3-875803a3fd66" volumeName="kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-client-ca-bundle" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529121    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="50e89fdd-eb0a-44c4-b904-9e7955f88504" volumeName="kubernetes.io/projected/50e89fdd-eb0a-44c4-b904-9e7955f88504-kube-api-access-7zv5n" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529132    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b24caade-1c7b-44e6-b04e-fbc0c44403fe" volumeName="kubernetes.io/projected/b24caade-1c7b-44e6-b04e-fbc0c44403fe-kube-api-access-zd8zx" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529143    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529156    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182c2fcb-792e-429f-9ab3-875803a3fd66" volumeName="kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-configmap-kubelet-serving-ca-bundle" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529168    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="182c2fcb-792e-429f-9ab3-875803a3fd66" volumeName="kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-client-certs" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529180    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1bf749d7-4d5c-4855-b945-fbf6b4953624" volumeName="kubernetes.io/secret/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529192    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ce18949-1971-4017-ab1c-5faa6d7412b2" volumeName="kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529204    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5fedc973-716d-4cdc-9b47-748582eba143" volumeName="kubernetes.io/empty-dir/5fedc973-716d-4cdc-9b47-748582eba143-data-volume" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529215    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7ad1f241-5476-4a43-a616-f26eff05c43d" volumeName="kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529278    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b3876106-cc95-464e-9f2c-be6504f26585" volumeName="kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-cni-binary-copy" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529292    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" volumeName="kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529303    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="03ed77f9-5e31-428a-988c-7811b60e299a" volumeName="kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-grpc-tls" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529314    6000 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5a1b5632-bc8c-4692-8e60-816ef5b35575" volumeName="kubernetes.io/projected/5a1b5632-bc8c-4692-8e60-816ef5b35575-kube-api-access-rlf4r" seLinuxMountContext=""
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529325    6000 reconstruct.go:97] "Volume reconstruction finished"
Nov 03 00:06:02.531712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.529331    6000 reconciler.go:29] "Reconciler: start to sync state"
Nov 03 00:06:02.556443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.556385    6000 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 03 00:06:02.558704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.558674    6000 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 03 00:06:02.558853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.558723    6000 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 03 00:06:02.558853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.558779    6000 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 03 00:06:02.558956 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:02.558827    6000 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 03 00:06:02.575789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.575754    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6" exitCode=0
Nov 03 00:06:02.575922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.575853    6000 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 03 00:06:02.576884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.576864    6000 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:06:02.586756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.586737    6000 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:06:02.588847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.588822    6000 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:06:02.590630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.590601    6000 generic.go:356] "Generic (PLEG): container finished" podID="0da10585-07cb-473c-94c2-d9a8c519ea73" containerID="801b11fdc26932b3ddb6da304d6c4bf1cd8b6a982ded042d7157abb390e3240a" exitCode=0
Nov 03 00:06:02.599150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.599128    6000 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 03 00:06:02.627496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.627464    6000 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="4763efdd5cc025115c2e2099db4cc46fde3fd1cfc032007ab5abd1af19043cff" exitCode=0
Nov 03 00:06:02.627496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.627492    6000 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="11920084ab51b4af7187bc2e0b6813f4ebc66675292f759fb26eebf427a1dfd7" exitCode=0
Nov 03 00:06:02.627496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.627502    6000 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="f6e4bc304b0e67b8e715fd8580182648cde62bdbf4c083d9aca396091458776a" exitCode=0
Nov 03 00:06:02.627701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.627509    6000 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="727362b073d706a7852af53e1e463a4a9924722ebbd938cf8e13d4b8d79cb027" exitCode=0
Nov 03 00:06:02.627701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.627519    6000 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="567a85ddaa1fb324ed598e190fbdbf0576523c7f6d6cf6cf0c669b2d0eb0a6c5" exitCode=0
Nov 03 00:06:02.627701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.627531    6000 generic.go:356] "Generic (PLEG): container finished" podID="e08c0805-8fa0-498f-a60c-9f24b8c6ddd2" containerID="4c750a2a345b8d0ac35d83d3537dfc73afa0e6f24b39f85c3212e5ad86bee93f" exitCode=0
Nov 03 00:06:02.629531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.629503    6000 generic.go:356] "Generic (PLEG): container finished" podID="3fc90fa10f5e4f787096a878d9a85228" containerID="d635ceaa75433184b8abf679a600e287e563ba1a6ad46dbfc3a39c249a3ef91a" exitCode=0
Nov 03 00:06:02.654377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.654320    6000 manager.go:324] Recovery completed
Nov 03 00:06:02.659012 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:02.658990    6000 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet"
Nov 03 00:06:02.680503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680482    6000 cpu_manager.go:226] "Starting" policy="none"
Nov 03 00:06:02.680503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680498    6000 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 03 00:06:02.680666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680515    6000 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 03 00:06:02.680666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680631    6000 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 03 00:06:02.680666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680641    6000 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 03 00:06:02.680666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680658    6000 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 03 00:06:02.680666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680663    6000 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 03 00:06:02.680666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680669    6000 policy_none.go:50] "Start"
Nov 03 00:06:02.680822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680678    6000 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 03 00:06:02.680822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680686    6000 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 03 00:06:02.680822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680804    6000 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 03 00:06:02.680822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680811    6000 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 03 00:06:02.680822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.680816    6000 policy_none.go:44] "Start"
Nov 03 00:06:02.685691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.685667    6000 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 03 00:06:02.688344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.688327    6000 manager.go:341] "Starting Device Plugin manager"
Nov 03 00:06:02.688921 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:02.688353    6000 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 03 00:06:02.689033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.688925    6000 server.go:87] "Starting device plugin registration server"
Nov 03 00:06:02.689174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.689164    6000 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 03 00:06:02.689208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.689175    6000 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 03 00:06:02.689316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.689296    6000 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 03 00:06:02.689441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.689402    6000 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 03 00:06:02.689441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.689443    6000 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 03 00:06:02.689671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.689450    6000 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 03 00:06:02.689671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.689503    6000 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T00:06:02.689433022Z","UUID":"7483608c-c21e-4090-8bfc-4a6eacac30d6","Handler":null,"Name":"","Endpoint":""}
Nov 03 00:06:02.691019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.690998    6000 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 03 00:06:02.691107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.691041    6000 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 03 00:06:02.691187 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:02.691164    6000 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 03 00:06:02.789637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.789599    6000 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 03 00:06:02.791770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.791748    6000 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientMemory"
Nov 03 00:06:02.791886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.791785    6000 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasNoDiskPressure"
Nov 03 00:06:02.791886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.791799    6000 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-136-97.ec2.internal" event="NodeHasSufficientPID"
Nov 03 00:06:02.791886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.791862    6000 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.823775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.823752    6000 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.823862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.823828    6000 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.859798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.859731    6000 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"]
Nov 03 00:06:02.860464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.860448    6000 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.860594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.860558    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"be3932b66fe25adccbe122883139881420ac0eb5bbf9813323661d8dc1b04798"}
Nov 03 00:06:02.860665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.860600    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal" event={"ID":"b6ab5ae10a035469fb072566acf5cdc5","Type":"ContainerStarted","Data":"ecee583e33fae8693d8213953e94cab4090b49cc51bf9119b79e1f6eaec1ad1d"}
Nov 03 00:06:02.860721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.860677    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"acca60ee61ec507f143dbc896b65cbc1bd6c37ead3acc5cce4e145206b1f3032"}
Nov 03 00:06:02.860721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.860691    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerDied","Data":"d635ceaa75433184b8abf679a600e287e563ba1a6ad46dbfc3a39c249a3ef91a"}
Nov 03 00:06:02.860721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.860709    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal" event={"ID":"3fc90fa10f5e4f787096a878d9a85228","Type":"ContainerStarted","Data":"38db531659de29fbc09904337d3247f8939df9f5156cbe5296a590663f81112c"}
Nov 03 00:06:02.860865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.860758    6000 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.882024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.881994    6000 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:06:02.882187 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:02.882064    6000 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.882454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.882438    6000 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:06:02.882493 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:02.882481    6000 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.977898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.977815    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.977898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.977855    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:02.977898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:02.977876    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.078429 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.078388    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.078429 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.078428    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.078633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.078446    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.078633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.078494    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.078633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.078504    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b6ab5ae10a035469fb072566acf5cdc5-config\") pod \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" (UID: \"b6ab5ae10a035469fb072566acf5cdc5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.078633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.078561    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/3fc90fa10f5e4f787096a878d9a85228-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" (UID: \"3fc90fa10f5e4f787096a878d9a85228\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.511191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.511154    6000 apiserver.go:52] "Watching apiserver"
Nov 03 00:06:03.561491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.561462    6000 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 03 00:06:03.562385 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.562356    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq","openshift-multus/multus-8kjlq","openshift-monitoring/metrics-server-c65fccf8-rt7bz","openshift-monitoring/alertmanager-main-0","openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs","openshift-multus/network-metrics-daemon-wjrvj","kube-system/global-pull-secret-syncer-xhsst","kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal","openshift-dns/node-resolver-6lccb","openshift-image-registry/node-ca-wf7kp","openshift-cluster-node-tuning-operator/tuned-k25z9","openshift-dns/dns-default-rv656","openshift-ovn-kubernetes/ovnkube-node-9dgcv","openshift-insights/insights-runtime-extractor-qc4b5","openshift-monitoring/prometheus-operator-9cd89b74c-srgkz","openshift-network-diagnostics/network-check-target-8l8m5","openshift-monitoring/node-exporter-ksp7h","openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv","kube-system/konnectivity-agent-9td9d","openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal","openshift-console/console-5fcfc8dbfc-dspqk","openshift-console/console-6c6c5cc84b-9bx56","openshift-ingress-canary/ingress-canary-9tx8l","openshift-multus/multus-additional-cni-plugins-bdnqz","openshift-network-operator/iptables-alerter-6w7xg","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq","openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"]
Nov 03 00:06:03.577463 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.577435    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.577632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.577561    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.577699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.577646    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2dzkq\""
Nov 03 00:06:03.577814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.577791    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9mnsd\""
Nov 03 00:06:03.577876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.577861    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.578291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.578247    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 03 00:06:03.578417 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.578397    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.578417 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.578414    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.578514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.578334    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 03 00:06:03.578514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.578440    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jtdbf\""
Nov 03 00:06:03.630215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.630185    6000 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 03 00:06:03.633765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.633744    6000 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.633879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.633871    6000 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.654024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.654003    6000 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:06:03.654085 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:03.654045    6000 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-136-97.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.654641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.654629    6000 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 03 00:06:03.654695 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:03.654684    6000 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-136-97.ec2.internal"
Nov 03 00:06:03.678378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678354    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.678515 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678401    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 03 00:06:03.678515 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678440    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 03 00:06:03.678515 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678451    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 03 00:06:03.678658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678526    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.678658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678625    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 03 00:06:03.678723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678677    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.678802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678785    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 03 00:06:03.678857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678823    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 03 00:06:03.678917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.678903    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.679057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679040    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.679177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679155    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 03 00:06:03.679177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679156    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 03 00:06:03.679369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679244    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.679440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679423    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 03 00:06:03.679504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679422    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.679504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679468    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 03 00:06:03.679504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679485    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.679644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679515    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 03 00:06:03.679644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679514    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.679644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679542    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 03 00:06:03.679644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679551    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.679644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679588    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.679844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679660    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.679844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679738    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-8v79z\""
Nov 03 00:06:03.679844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679747    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 03 00:06:03.679844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679756    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 03 00:06:03.679844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679792    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-gs4dg\""
Nov 03 00:06:03.680045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679905    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 03 00:06:03.680045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.679943    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 03 00:06:03.680116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680061    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 03 00:06:03.680231 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680209    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.680325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680246    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlf4r\" (UniqueName: \"kubernetes.io/projected/5a1b5632-bc8c-4692-8e60-816ef5b35575-kube-api-access-rlf4r\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.680325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680299    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-registration-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.680435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680337    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-sys\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.680435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680359    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-w8ktr\""
Nov 03 00:06:03.680435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680366    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-host\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.680435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680395    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhx6r\" (UniqueName: \"kubernetes.io/projected/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-kube-api-access-nhx6r\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:06:03.680615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680440    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.680615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680462    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 03 00:06:03.680615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680470    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 03 00:06:03.680615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680518    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.680615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680534    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xfsv5\""
Nov 03 00:06:03.680615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680608    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.680883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680670    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-etc-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.680883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680715    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.680883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680768    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.680883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680799    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.680883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680830    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9xxv\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.680883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680850    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680920    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-cert\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680948    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-sys\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680965    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680980    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zglcx\" (UniqueName: \"kubernetes.io/projected/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-kube-api-access-zglcx\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.680996    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fedc973-716d-4cdc-9b47-748582eba143-crio-socket\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681012    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzz28\" (UniqueName: \"kubernetes.io/projected/5fedc973-716d-4cdc-9b47-748582eba143-kube-api-access-vzz28\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681030    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-host-slash\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681072    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s66w5\" (UniqueName: \"kubernetes.io/projected/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-kube-api-access-s66w5\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.681124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681075    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.681491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681190    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-var-lib-kubelet\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.681491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681292    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.681491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681331    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.681491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681410    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-kubelet-config\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.681491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681433    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:06:03.681491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681449    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-device-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.681491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681480    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.681749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681550    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:06:03.681749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681646    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n88np\" (UniqueName: \"kubernetes.io/projected/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-api-access-n88np\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.681749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681711    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfgmh\" (UniqueName: \"kubernetes.io/projected/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-kube-api-access-lfgmh\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:06:03.681749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681730    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.681749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681748    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-multus-daemon-config\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.681996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681771    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.681996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681813    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxxbd\" (UniqueName: \"kubernetes.io/projected/3ce18949-1971-4017-ab1c-5faa6d7412b2-kube-api-access-zxxbd\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:06:03.681996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681834    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-metrics-client-ca\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.681996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681862    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrnc\" (UniqueName: \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.681996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681887    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.681996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.681989    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-multus-daemon-config\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682009    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682038    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-tmp\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682054    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682073    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682087    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682139    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-federate-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682150    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682168    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-log-socket\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682186    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-tmp\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.682242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682197    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-netd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682307    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682315    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682336    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682381    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjr79\" (UniqueName: \"kubernetes.io/projected/b3876106-cc95-464e-9f2c-be6504f26585-kube-api-access-hjr79\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682424    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682462    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682501    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682542    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd8zx\" (UniqueName: \"kubernetes.io/projected/b24caade-1c7b-44e6-b04e-fbc0c44403fe-kube-api-access-zd8zx\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682548    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682622    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682654    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682620    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-metrics-tls\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682668    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 03 00:06:03.682697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682683    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682774    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1b5632-bc8c-4692-8e60-816ef5b35575-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682808    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-multus-certs\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682836    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-metrics-tls\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682837    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682888    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682912    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-tls\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682929    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-client-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682949    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.682978    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba3da01-3b60-48ec-8d2d-2d90478e334a-host\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683004    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683038    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683071    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.683110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683105    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683139    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683162    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683276    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-cni-binary-copy\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683296    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cni-binary-copy\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683303    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-k8s-cni-cncf-io\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683361    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-hostroot\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683390    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-etc-kubernetes\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683416    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.683484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683439    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b3876106-cc95-464e-9f2c-be6504f26585-cni-binary-copy\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683541    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-dbus\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683564    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-conf\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683565    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683584    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-root\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683611    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-accelerators-collector-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683636    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-metrics-server-audit-profiles\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683667    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683686    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59sh\" (UniqueName: \"kubernetes.io/projected/fba3da01-3b60-48ec-8d2d-2d90478e334a-kube-api-access-l59sh\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683703    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j7s6\" (UniqueName: \"kubernetes.io/projected/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-kube-api-access-9j7s6\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:06:03.683726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683719    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-bin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683735    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683839    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683877    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-config\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683881    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/182c2fcb-792e-429f-9ab3-875803a3fd66-metrics-server-audit-profiles\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683879    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683910    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683952    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.683994    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-ovn\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684021    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysconfig\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684046    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-systemd\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684079    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-config\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684090    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-tmp-dir\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684140    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-tmp-dir\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.684179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684163    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-tmp-dir\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684192    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684299    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-systemd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684318    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684334    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684337    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684349    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-modprobe-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684360    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-tmp-dir\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684390    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684427    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fedc973-716d-4cdc-9b47-748582eba143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684454    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-socket-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684471    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684486    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-socket-dir-parent\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684540    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-agent-certs\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684563    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-hosts-file\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684598    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684622    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684642    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpp5m\" (UniqueName: \"kubernetes.io/projected/03ed77f9-5e31-428a-988c-7811b60e299a-kube-api-access-vpp5m\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684659    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-config-volume\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684707    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-systemd-units\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684729    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684733    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-bin\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.684754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684765    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684799    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684848    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fba3da01-3b60-48ec-8d2d-2d90478e334a-serviceca\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684874    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-kubelet\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684911    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684912    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-sys-fs\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684945    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-iptables-alerter-script\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684952    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684970    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cnibin\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.684994    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-netns\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685020    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-kubernetes\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685039    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-client-certs\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685055    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-netns\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685071    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-konnectivity-ca\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685087    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685105    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-wtmp\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685117    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-iptables-alerter-script\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685214    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5z47\" (UniqueName: \"kubernetes.io/projected/0da10585-07cb-473c-94c2-d9a8c519ea73-kube-api-access-v5z47\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685267    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-client-certs\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685286    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2w2j6\" (UniqueName: \"kubernetes.io/projected/182c2fcb-792e-429f-9ab3-875803a3fd66-kube-api-access-2w2j6\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685305    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-konnectivity-ca\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685371    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-var-lib-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685458    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fedc973-716d-4cdc-9b47-748582eba143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685502    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgdjp\" (UniqueName: \"kubernetes.io/projected/9df7b833-c25f-48d0-8de2-401b05bd6796-kube-api-access-rgdjp\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685533    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/fba3da01-3b60-48ec-8d2d-2d90478e334a-serviceca\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685545    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-kubelet\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685578    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685598    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/5fedc973-716d-4cdc-9b47-748582eba143-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685611    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-multus\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685646    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685691    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685715    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685733    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.685754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685763    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685791    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-run\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685817    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-tls\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685851    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-serving-certs-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685882    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685913    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685938    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685949    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685955    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-cnibin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.685995    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ed77f9-5e31-428a-988c-7811b60e299a-metrics-client-ca\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686026    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686032    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-serving-certs-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686052    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-env-overrides\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686080    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686101    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-textfile\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686119    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686135    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686164    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ed77f9-5e31-428a-988c-7811b60e299a-tmp\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686179    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-env-overrides\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686188    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-textfile\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686191    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/182c2fcb-792e-429f-9ab3-875803a3fd66-audit-log\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686239    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686278    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/03ed77f9-5e31-428a-988c-7811b60e299a-tmp\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686281    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-node-log\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686311    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686319    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/182c2fcb-792e-429f-9ab3-875803a3fd66-audit-log\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686334    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686339    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h6zb\" (UniqueName: \"kubernetes.io/projected/1bf749d7-4d5c-4855-b945-fbf6b4953624-kube-api-access-9h6zb\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686371    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fedc973-716d-4cdc-9b47-748582eba143-data-volume\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686457    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686488    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjv8x\" (UniqueName: \"kubernetes.io/projected/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-kube-api-access-mjv8x\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686515    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zv5n\" (UniqueName: \"kubernetes.io/projected/50e89fdd-eb0a-44c4-b904-9e7955f88504-kube-api-access-7zv5n\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686532    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/5fedc973-716d-4cdc-9b47-748582eba143-data-volume\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686571    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-system-cni-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686650    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-system-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686678    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-conf-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686690    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/5a1b5632-bc8c-4692-8e60-816ef5b35575-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686699    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686725    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-tuned\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686763    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686826    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686835    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-tuned\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686861    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wssr\" (UniqueName: \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686892    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.686898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686909    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686936    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-grpc-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686977    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-slash\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.686985    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.687004    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-script-lib\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.687043    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-lib-modules\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.687072    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-metrics-client-ca\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.687075    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.687089    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.688090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.687163    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckjgg\" (UniqueName: \"kubernetes.io/projected/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-kube-api-access-ckjgg\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.689607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.689592    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 03 00:06:03.689607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.689601    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qr8z8\""
Nov 03 00:06:03.690405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.690393    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-dlnxq\""
Nov 03 00:06:03.690454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.690406    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.690454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.690418    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.690549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.690474    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-nrnbx\""
Nov 03 00:06:03.690549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.690525    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 03 00:06:03.691000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.690986    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 03 00:06:03.691110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691091    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovn-node-metrics-cert\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.691162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691122    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691231    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691244    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691237    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691273    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691473    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691479    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691488    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691480    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691510    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691548    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691551    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691574    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691578    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691571    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691592    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vhndk\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691599    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-56fpg\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691628    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691664    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691677    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691692    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691753    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691753    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691855    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-pr6ld\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691863    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691855    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-wsqcw\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691876    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-gt4zj\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691928    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691932    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691941    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691927    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3ce18949-1971-4017-ab1c-5faa6d7412b2-metrics-certs\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691972    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-7b8cw\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.691975    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-ct6nh\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692009    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqxtt\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692022    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-rrqd5\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692026    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692049    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692201    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692152    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-j9n2q\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692292    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692308    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692353    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-xdt4r\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692385    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692453    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tbmwn\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692489    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-federate-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692494    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9s6uds6316t24\""
Nov 03 00:06:03.692536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692547    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692580    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-4gj81ut6di7s2\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692702    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692834    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692847    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7llq5\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.692937    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693015    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-5qw2l\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693040    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693073    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5a1b5632-bc8c-4692-8e60-816ef5b35575-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693091    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-rgf75\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693220    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-client-ca-bundle\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693219    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/182c2fcb-792e-429f-9ab3-875803a3fd66-secret-metrics-server-tls\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693352    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693428    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.693436    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694046    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Nov 03 00:06:03.694216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694127    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.694939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694322    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-accelerators-collector-config\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.694939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694413    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.694939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694610    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.694939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694814    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/21ef2aeb-b54a-45f5-b58a-af731ebdfe11-agent-certs\") pod \"konnectivity-agent-9td9d\" (UID: \"21ef2aeb-b54a-45f5-b58a-af731ebdfe11\") " pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:06:03.694939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694734    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.695190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.694993    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-config-volume\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.695190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.695120    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/5fedc973-716d-4cdc-9b47-748582eba143-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.695190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.695154    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.695428 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.695406    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 03 00:06:03.698425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698404    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.698514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698449    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.698514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698473    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0da10585-07cb-473c-94c2-d9a8c519ea73-metrics-client-ca\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.698514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698482    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.698514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698452    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-client-tls\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.698514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698485    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1bf749d7-4d5c-4855-b945-fbf6b4953624-ovnkube-script-lib\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698486    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-tls\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698638    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698641    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/03ed77f9-5e31-428a-988c-7811b60e299a-metrics-client-ca\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698675    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698722    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-thanos-querier-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698727    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698748    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 00:06:03.698798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.698750    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/03ed77f9-5e31-428a-988c-7811b60e299a-secret-grpc-tls\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.699306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.699289    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.700313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.700295    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxxbd\" (UniqueName: \"kubernetes.io/projected/3ce18949-1971-4017-ab1c-5faa6d7412b2-kube-api-access-zxxbd\") pod \"network-metrics-daemon-wjrvj\" (UID: \"3ce18949-1971-4017-ab1c-5faa6d7412b2\") " pod="openshift-multus/network-metrics-daemon-wjrvj"
Nov 03 00:06:03.700454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.700437    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfgmh\" (UniqueName: \"kubernetes.io/projected/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-kube-api-access-lfgmh\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:06:03.700678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.700656    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlf4r\" (UniqueName: \"kubernetes.io/projected/5a1b5632-bc8c-4692-8e60-816ef5b35575-kube-api-access-rlf4r\") pod \"prometheus-operator-9cd89b74c-srgkz\" (UID: \"5a1b5632-bc8c-4692-8e60-816ef5b35575\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-srgkz"
Nov 03 00:06:03.700745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.700736    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zglcx\" (UniqueName: \"kubernetes.io/projected/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-kube-api-access-zglcx\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:06:03.700799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.700753    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.700885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.700865    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s66w5\" (UniqueName: \"kubernetes.io/projected/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-kube-api-access-s66w5\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.700954 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.700940    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7ad1f241-5476-4a43-a616-f26eff05c43d-original-pull-secret\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.701069 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.701053    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.701297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.701278    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhx6r\" (UniqueName: \"kubernetes.io/projected/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-kube-api-access-nhx6r\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:06:03.701331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.701287    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f-cert\") pod \"ingress-canary-9tx8l\" (UID: \"79ae98d8-e66e-4d26-ba5e-d3e0f0d9421f\") " pod="openshift-ingress-canary/ingress-canary-9tx8l"
Nov 03 00:06:03.701571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.701556    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.701690 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.701673    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n88np\" (UniqueName: \"kubernetes.io/projected/dcd7e2a0-6e98-4417-bbc9-25868b1ca911-kube-api-access-n88np\") pod \"kube-state-metrics-5dc8d7d89b-2q7ws\" (UID: \"dcd7e2a0-6e98-4417-bbc9-25868b1ca911\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-2q7ws"
Nov 03 00:06:03.701746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.701681    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-nv4rq\" (UID: \"4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:06:03.701861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.701845    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzz28\" (UniqueName: \"kubernetes.io/projected/5fedc973-716d-4cdc-9b47-748582eba143-kube-api-access-vzz28\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.702057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.702043    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/50e89fdd-eb0a-44c4-b904-9e7955f88504-metrics-client-ca\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.702057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.702050    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.702156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.702107    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9xxv\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.704492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.704473    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd8zx\" (UniqueName: \"kubernetes.io/projected/b24caade-1c7b-44e6-b04e-fbc0c44403fe-kube-api-access-zd8zx\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.704683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.704665    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrnc\" (UniqueName: \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\") pod \"console-5fcfc8dbfc-dspqk\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") " pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:03.705107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.705091    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j7s6\" (UniqueName: \"kubernetes.io/projected/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-kube-api-access-9j7s6\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:06:03.705867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.705848    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjr79\" (UniqueName: \"kubernetes.io/projected/b3876106-cc95-464e-9f2c-be6504f26585-kube-api-access-hjr79\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.705943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.705864    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59sh\" (UniqueName: \"kubernetes.io/projected/fba3da01-3b60-48ec-8d2d-2d90478e334a-kube-api-access-l59sh\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:06:03.706833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.706817    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:03.708073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.708052    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpp5m\" (UniqueName: \"kubernetes.io/projected/03ed77f9-5e31-428a-988c-7811b60e299a-kube-api-access-vpp5m\") pod \"thanos-querier-6bbb79ccc9-zpjcf\" (UID: \"03ed77f9-5e31-428a-988c-7811b60e299a\") " pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:03.708162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.708146    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgdjp\" (UniqueName: \"kubernetes.io/projected/9df7b833-c25f-48d0-8de2-401b05bd6796-kube-api-access-rgdjp\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.708440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.708425    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5z47\" (UniqueName: \"kubernetes.io/projected/0da10585-07cb-473c-94c2-d9a8c519ea73-kube-api-access-v5z47\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.708968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.708951    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zv5n\" (UniqueName: \"kubernetes.io/projected/50e89fdd-eb0a-44c4-b904-9e7955f88504-kube-api-access-7zv5n\") pod \"telemeter-client-6b45cfff7c-f6ktv\" (UID: \"50e89fdd-eb0a-44c4-b904-9e7955f88504\") " pod="openshift-monitoring/telemeter-client-6b45cfff7c-f6ktv"
Nov 03 00:06:03.709283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.709243    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h6zb\" (UniqueName: \"kubernetes.io/projected/1bf749d7-4d5c-4855-b945-fbf6b4953624-kube-api-access-9h6zb\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.709339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.709285    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckjgg\" (UniqueName: \"kubernetes.io/projected/8c3c97a4-e610-4a34-9fe8-b5e754372cd2-kube-api-access-ckjgg\") pod \"dns-default-rv656\" (UID: \"8c3c97a4-e610-4a34-9fe8-b5e754372cd2\") " pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:03.709339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.709301    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2w2j6\" (UniqueName: \"kubernetes.io/projected/182c2fcb-792e-429f-9ab3-875803a3fd66-kube-api-access-2w2j6\") pod \"metrics-server-c65fccf8-rt7bz\" (UID: \"182c2fcb-792e-429f-9ab3-875803a3fd66\") " pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:03.710248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.710232    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wssr\" (UniqueName: \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\") pod \"console-6c6c5cc84b-9bx56\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") " pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:03.710658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.710639    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xp29\" (UniqueName: \"kubernetes.io/projected/04183b41-8425-43cd-a145-9e454673cad1-kube-api-access-8xp29\") pod \"network-check-target-8l8m5\" (UID: \"04183b41-8425-43cd-a145-9e454673cad1\") " pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:06:03.710742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.710728    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjv8x\" (UniqueName: \"kubernetes.io/projected/a2de66e0-f1f5-44d4-8eca-3bd623d6fd84-kube-api-access-mjv8x\") pod \"openshift-state-metrics-7bd7489957-df6xs\" (UID: \"a2de66e0-f1f5-44d4-8eca-3bd623d6fd84\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-df6xs"
Nov 03 00:06:03.787978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.787897    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-bin\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.787978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.787927    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-kubelet\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.787978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.787944    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-sys-fs\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.787978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.787960    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cnibin\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.787978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.787973    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-netns\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788017    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-cnibin\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788028    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-kubelet\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788034    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-bin\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788038    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-sys-fs\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788046    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-kubernetes\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788070    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-netns\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788086    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-kubernetes\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788104    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-netns\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788125    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-wtmp\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788142    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-var-lib-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788159    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-kubelet\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788191    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-var-lib-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788196    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-kubelet\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788203    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-netns\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788209    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-multus\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788227    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-run\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788235    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-multus\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788195    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-node-exporter-wtmp\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788249    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-cnibin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788299    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-cnibin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788303    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-run\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788326    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-node-log\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788348    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788365    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-system-cni-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788380    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-system-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788394    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-run-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.788389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788402    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-conf-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788419    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-node-log\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788427    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788433    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-conf-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788402    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-system-cni-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788453    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-system-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788456    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-slash\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788485    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-slash\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788486    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-lib-modules\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788514    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-registration-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788534    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-sys\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788546    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788555    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-host\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788569    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-registration-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788575    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-sys\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788580    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-host\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788581    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-etc-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788558    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-lib-modules\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788605    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-sys\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788620    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fedc973-716d-4cdc-9b47-748582eba143-crio-socket\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788638    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-host-slash\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788658    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-var-lib-kubelet\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788661    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-sys\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788675    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-kubelet-config\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788689    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-device-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788690    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-etc-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788701    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/5fedc973-716d-4cdc-9b47-748582eba143-crio-socket\") pod \"insights-runtime-extractor-qc4b5\" (UID: \"5fedc973-716d-4cdc-9b47-748582eba143\") " pod="openshift-insights/insights-runtime-extractor-qc4b5"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788722    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-log-socket\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788736    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-device-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788742    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/83c6f1f9-3e2f-4c6e-801f-29735ad355a2-host-slash\") pod \"iptables-alerter-6w7xg\" (UID: \"83c6f1f9-3e2f-4c6e-801f-29735ad355a2\") " pod="openshift-network-operator/iptables-alerter-6w7xg"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788738    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-netd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788763    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-cni-netd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788770    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-log-socket\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788769    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-var-lib-kubelet\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788762    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-kubelet-config\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788779    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-multus-certs\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788835    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba3da01-3b60-48ec-8d2d-2d90478e334a-host\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788863    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-multus-certs\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788866    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788872    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/fba3da01-3b60-48ec-8d2d-2d90478e334a-host\") pod \"node-ca-wf7kp\" (UID: \"fba3da01-3b60-48ec-8d2d-2d90478e334a\") " pod="openshift-image-registry/node-ca-wf7kp"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788887    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788905    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788925    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-k8s-cni-cncf-io\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788948    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-hostroot\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788969    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-etc-kubernetes\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788973    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-cni-dir\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788964    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788991    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-openvswitch\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788991    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-run-k8s-cni-cncf-io\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789010    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-hostroot\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.788991    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-dbus\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789045    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7ad1f241-5476-4a43-a616-f26eff05c43d-dbus\") pod \"global-pull-secret-syncer-xhsst\" (UID: \"7ad1f241-5476-4a43-a616-f26eff05c43d\") " pod="kube-system/global-pull-secret-syncer-xhsst"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789049    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-conf\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789013    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-etc-kubernetes\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789071    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-root\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789100    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0da10585-07cb-473c-94c2-d9a8c519ea73-root\") pod \"node-exporter-ksp7h\" (UID: \"0da10585-07cb-473c-94c2-d9a8c519ea73\") " pod="openshift-monitoring/node-exporter-ksp7h"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789101    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-bin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789132    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789160    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-ovn\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789183    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysconfig\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789194    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysctl-conf\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789206    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-systemd\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789212    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-host-var-lib-cni-bin\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789217    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-kubelet-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789227    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-ovn\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789238    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-systemd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789269    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-sysconfig\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789294    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789312    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-modprobe-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789327    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-socket-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789341    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789242    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-systemd\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789294    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-run-systemd\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789365    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-etc-selinux\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789379    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-socket-dir-parent\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789381    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9df7b833-c25f-48d0-8de2-401b05bd6796-socket-dir\") pod \"aws-ebs-csi-driver-node-wk4vq\" (UID: \"9df7b833-c25f-48d0-8de2-401b05bd6796\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-wk4vq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789394    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e08c0805-8fa0-498f-a60c-9f24b8c6ddd2-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bdnqz\" (UID: \"e08c0805-8fa0-498f-a60c-9f24b8c6ddd2\") " pod="openshift-multus/multus-additional-cni-plugins-bdnqz"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789389    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b24caade-1c7b-44e6-b04e-fbc0c44403fe-etc-modprobe-d\") pod \"tuned-k25z9\" (UID: \"b24caade-1c7b-44e6-b04e-fbc0c44403fe\") " pod="openshift-cluster-node-tuning-operator/tuned-k25z9"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789415    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-hosts-file\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789437    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-systemd-units\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789462    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0-hosts-file\") pod \"node-resolver-6lccb\" (UID: \"cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0\") " pod="openshift-dns/node-resolver-6lccb"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789463    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b3876106-cc95-464e-9f2c-be6504f26585-multus-socket-dir-parent\") pod \"multus-8kjlq\" (UID: \"b3876106-cc95-464e-9f2c-be6504f26585\") " pod="openshift-multus/multus-8kjlq"
Nov 03 00:06:03.789522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:03.789489    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1bf749d7-4d5c-4855-b945-fbf6b4953624-systemd-units\") pod \"ovnkube-node-9dgcv\" (UID: \"1bf749d7-4d5c-4855-b945-fbf6b4953624\") " pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:06.130560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:06.130532    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:06.280537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:06.280509    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-nv4rq"
Nov 03 00:06:06.506290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:06.506212    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rv656"
Nov 03 00:06:06.884192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:06.884166    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:06.884431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:06.884418    6000 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 03 00:06:06.956818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:06.956787    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-6bbb79ccc9-zpjcf"
Nov 03 00:06:08.983820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:08.983769    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:06:11.551991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:11.551955    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:06:14.975175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:14.975099    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9td9d"
Nov 03 00:06:15.648325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:15.648293    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-c65fccf8-rt7bz"
Nov 03 00:06:19.894149 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:19.894111    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:06:28.593795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.593738    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6c6c5cc84b-9bx56" podUID="32ce8d43-d4b6-4b23-a235-624a0e30612d" containerName="console" containerID="cri-o://1392cd07b34b5073656227d54c10341d224e0e7696da2d9d920a1b99376648b8" gracePeriod=15
Nov 03 00:06:28.729729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.729702    6000 generic.go:356] "Generic (PLEG): container finished" podID="32ce8d43-d4b6-4b23-a235-624a0e30612d" containerID="1392cd07b34b5073656227d54c10341d224e0e7696da2d9d920a1b99376648b8" exitCode=0
Nov 03 00:06:28.729868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.729772    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c5cc84b-9bx56" event={"ID":"32ce8d43-d4b6-4b23-a235-624a0e30612d","Type":"ContainerDied","Data":"1392cd07b34b5073656227d54c10341d224e0e7696da2d9d920a1b99376648b8"}
Nov 03 00:06:28.841050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.841030    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:28.909610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.909576    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\") pod \"32ce8d43-d4b6-4b23-a235-624a0e30612d\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") "
Nov 03 00:06:28.909779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.909660    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\" (UniqueName: \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\") pod \"32ce8d43-d4b6-4b23-a235-624a0e30612d\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") "
Nov 03 00:06:28.909779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.909732    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\" (UniqueName: \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\") pod \"32ce8d43-d4b6-4b23-a235-624a0e30612d\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") "
Nov 03 00:06:28.909779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.909762    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\") pod \"32ce8d43-d4b6-4b23-a235-624a0e30612d\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") "
Nov 03 00:06:28.909940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.909802    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\") pod \"32ce8d43-d4b6-4b23-a235-624a0e30612d\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") "
Nov 03 00:06:28.909940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.909843    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\") pod \"32ce8d43-d4b6-4b23-a235-624a0e30612d\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") "
Nov 03 00:06:28.909940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.909875    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\") pod \"32ce8d43-d4b6-4b23-a235-624a0e30612d\" (UID: \"32ce8d43-d4b6-4b23-a235-624a0e30612d\") "
Nov 03 00:06:28.910099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.910050    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp" pod "32ce8d43-d4b6-4b23-a235-624a0e30612d" (UID: "32ce8d43-d4b6-4b23-a235-624a0e30612d"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:06:28.910174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.910057    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca" pod "32ce8d43-d4b6-4b23-a235-624a0e30612d" (UID: "32ce8d43-d4b6-4b23-a235-624a0e30612d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:28.910232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.910186    6000 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32ce8d43-d4b6-4b23-a235-624a0e30612d-tmp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:28.910362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.910333    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config" pod "32ce8d43-d4b6-4b23-a235-624a0e30612d" (UID: "32ce8d43-d4b6-4b23-a235-624a0e30612d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:28.910439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.910417    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert" pod "32ce8d43-d4b6-4b23-a235-624a0e30612d" (UID: "32ce8d43-d4b6-4b23-a235-624a0e30612d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:28.912276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.912228    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr" pod "32ce8d43-d4b6-4b23-a235-624a0e30612d" (UID: "32ce8d43-d4b6-4b23-a235-624a0e30612d"). InnerVolumeSpecName "kube-api-access-7wssr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:28.912276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.912232    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert" pod "32ce8d43-d4b6-4b23-a235-624a0e30612d" (UID: "32ce8d43-d4b6-4b23-a235-624a0e30612d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:28.912427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:28.912329    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config" pod "32ce8d43-d4b6-4b23-a235-624a0e30612d" (UID: "32ce8d43-d4b6-4b23-a235-624a0e30612d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:29.010688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.010655    6000 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-oauth-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:29.010688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.010684    6000 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-oauth-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:29.010866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.010700    6000 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:29.010866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.010712    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wssr\" (UniqueName: \"kubernetes.io/projected/32ce8d43-d4b6-4b23-a235-624a0e30612d-kube-api-access-7wssr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:29.010866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.010725    6000 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:29.010866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.010736    6000 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/32ce8d43-d4b6-4b23-a235-624a0e30612d-console-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:29.734433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.734399    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c6c5cc84b-9bx56" event={"ID":"32ce8d43-d4b6-4b23-a235-624a0e30612d","Type":"ContainerDied","Data":"46a2bef15f0cdac58de47c4b5786f10866fa4697a3a301195f868d2713aa80f6"}
Nov 03 00:06:29.734433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.734418    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c6c5cc84b-9bx56"
Nov 03 00:06:29.734889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.734461    6000 scope.go:122] "RemoveContainer" containerID="1392cd07b34b5073656227d54c10341d224e0e7696da2d9d920a1b99376648b8"
Nov 03 00:06:29.782629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.782596    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c6c5cc84b-9bx56"]
Nov 03 00:06:29.794249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:29.794226    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c6c5cc84b-9bx56"]
Nov 03 00:06:30.563775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:30.563745    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ce8d43-d4b6-4b23-a235-624a0e30612d" path="/var/lib/kubelet/pods/32ce8d43-d4b6-4b23-a235-624a0e30612d/volumes"
Nov 03 00:06:33.616047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:33.615999    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9dgcv"
Nov 03 00:06:38.986723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:38.986692    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8l8m5"
Nov 03 00:06:49.893934 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:49.893892    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:06:52.127848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.127815    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:06:52.128602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.128117    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="alertmanager" containerID="cri-o://19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547" gracePeriod=120
Nov 03 00:06:52.128602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.128201    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-web" containerID="cri-o://65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a" gracePeriod=120
Nov 03 00:06:52.128602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.128199    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-metric" containerID="cri-o://316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4" gracePeriod=120
Nov 03 00:06:52.128602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.128246    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy" containerID="cri-o://6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019" gracePeriod=120
Nov 03 00:06:52.128602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.128310    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="config-reloader" containerID="cri-o://2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac" gracePeriod=120
Nov 03 00:06:52.128602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.128295    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="prom-label-proxy" containerID="cri-o://abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5" gracePeriod=120
Nov 03 00:06:52.819282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819233    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5" exitCode=0
Nov 03 00:06:52.819282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819276    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019" exitCode=0
Nov 03 00:06:52.819282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819284    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac" exitCode=0
Nov 03 00:06:52.819516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819293    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547" exitCode=0
Nov 03 00:06:52.819516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819306    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"}
Nov 03 00:06:52.819516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819367    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"}
Nov 03 00:06:52.819516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819392    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"}
Nov 03 00:06:52.819516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:52.819417    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"}
Nov 03 00:06:53.369142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.369120    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:53.514166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514089    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514129    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514154    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514187    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514232    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514285    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514320    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514346    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514392    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514441    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514471    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514503    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514538    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\") pod \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\" (UID: \"e8b2a49b-6679-40ce-91f0-91fcfd7c7118\") "
Nov 03 00:06:53.514903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514644    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:53.514903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514830    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-metrics-client-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.514903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.514930    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:06:53.515753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.515726    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:06:53.517621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.517580    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:53.518068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.518029    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:06:53.518470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.518431    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:53.518635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.518606    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:53.518635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.518620    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "kube-api-access-l9xxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:53.518792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.518776    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:53.519141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.519114    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:53.519286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.519238    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:06:53.522828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.522806    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:53.528865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.528847    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config" pod "e8b2a49b-6679-40ce-91f0-91fcfd7c7118" (UID: "e8b2a49b-6679-40ce-91f0-91fcfd7c7118"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:06:53.615532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615510    6000 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-main-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615532    6000 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615543    6000 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-main-db\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615552    6000 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-tls-assets\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615560    6000 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615590    6000 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-web-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615599    6000 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615606    6000 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-config-out\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615614    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9xxv\" (UniqueName: \"kubernetes.io/projected/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-kube-api-access-l9xxv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615622    6000 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615630    6000 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.615657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.615640    6000 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/e8b2a49b-6679-40ce-91f0-91fcfd7c7118-cluster-tls-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:06:53.825802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.825720    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4" exitCode=0
Nov 03 00:06:53.825802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.825749    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerID="65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a" exitCode=0
Nov 03 00:06:53.825802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.825779    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"}
Nov 03 00:06:53.825999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.825820    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:53.825999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.825852    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"}
Nov 03 00:06:53.825999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.825882    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"e8b2a49b-6679-40ce-91f0-91fcfd7c7118","Type":"ContainerDied","Data":"bcc200aad21f6050cbd85702e8529d4c4deac0a1f896c951ea8fc26576e4a433"}
Nov 03 00:06:53.825999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.825913    6000 scope.go:122] "RemoveContainer" containerID="abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"
Nov 03 00:06:53.833959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.833944    6000 scope.go:122] "RemoveContainer" containerID="316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"
Nov 03 00:06:53.841157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.841134    6000 scope.go:122] "RemoveContainer" containerID="6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"
Nov 03 00:06:53.847833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.847817    6000 scope.go:122] "RemoveContainer" containerID="65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"
Nov 03 00:06:53.854119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.854102    6000 scope.go:122] "RemoveContainer" containerID="2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"
Nov 03 00:06:53.860669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.860652    6000 scope.go:122] "RemoveContainer" containerID="19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"
Nov 03 00:06:53.867728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.867711    6000 scope.go:122] "RemoveContainer" containerID="432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6"
Nov 03 00:06:53.874400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.874384    6000 scope.go:122] "RemoveContainer" containerID="abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"
Nov 03 00:06:53.874696 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:53.874667    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5\": container with ID starting with abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5 not found: ID does not exist" containerID="abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"
Nov 03 00:06:53.874757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.874705    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"} err="failed to get container status \"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5\": rpc error: code = NotFound desc = could not find container \"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5\": container with ID starting with abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5 not found: ID does not exist"
Nov 03 00:06:53.874757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.874738    6000 scope.go:122] "RemoveContainer" containerID="316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"
Nov 03 00:06:53.874979 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:53.874965    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4\": container with ID starting with 316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4 not found: ID does not exist" containerID="316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"
Nov 03 00:06:53.875020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.874983    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"} err="failed to get container status \"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4\": rpc error: code = NotFound desc = could not find container \"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4\": container with ID starting with 316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4 not found: ID does not exist"
Nov 03 00:06:53.875020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.874995    6000 scope.go:122] "RemoveContainer" containerID="6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"
Nov 03 00:06:53.875207 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:53.875191    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019\": container with ID starting with 6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019 not found: ID does not exist" containerID="6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"
Nov 03 00:06:53.875246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.875221    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"} err="failed to get container status \"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019\": rpc error: code = NotFound desc = could not find container \"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019\": container with ID starting with 6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019 not found: ID does not exist"
Nov 03 00:06:53.875246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.875237    6000 scope.go:122] "RemoveContainer" containerID="65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"
Nov 03 00:06:53.875470 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:53.875452    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a\": container with ID starting with 65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a not found: ID does not exist" containerID="65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"
Nov 03 00:06:53.875566 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.875472    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"} err="failed to get container status \"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a\": rpc error: code = NotFound desc = could not find container \"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a\": container with ID starting with 65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a not found: ID does not exist"
Nov 03 00:06:53.875566 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.875492    6000 scope.go:122] "RemoveContainer" containerID="2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"
Nov 03 00:06:53.875762 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:53.875746    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac\": container with ID starting with 2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac not found: ID does not exist" containerID="2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"
Nov 03 00:06:53.875811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.875765    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"} err="failed to get container status \"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac\": rpc error: code = NotFound desc = could not find container \"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac\": container with ID starting with 2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac not found: ID does not exist"
Nov 03 00:06:53.875811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.875777    6000 scope.go:122] "RemoveContainer" containerID="19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"
Nov 03 00:06:53.876004 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:53.875982    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547\": container with ID starting with 19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547 not found: ID does not exist" containerID="19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"
Nov 03 00:06:53.876071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876008    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"} err="failed to get container status \"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547\": rpc error: code = NotFound desc = could not find container \"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547\": container with ID starting with 19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547 not found: ID does not exist"
Nov 03 00:06:53.876071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876024    6000 scope.go:122] "RemoveContainer" containerID="432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6"
Nov 03 00:06:53.876378 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:06:53.876358    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6\": container with ID starting with 432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6 not found: ID does not exist" containerID="432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6"
Nov 03 00:06:53.876427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876383    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6"} err="failed to get container status \"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6\": rpc error: code = NotFound desc = could not find container \"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6\": container with ID starting with 432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6 not found: ID does not exist"
Nov 03 00:06:53.876427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876394    6000 scope.go:122] "RemoveContainer" containerID="abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"
Nov 03 00:06:53.876644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876627    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5"} err="failed to get container status \"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5\": rpc error: code = NotFound desc = could not find container \"abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5\": container with ID starting with abb4dca603cb4c3c46bfe1ecd7570dbdcadd423f74b6a54961bf455b760d4dd5 not found: ID does not exist"
Nov 03 00:06:53.876644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876644    6000 scope.go:122] "RemoveContainer" containerID="316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"
Nov 03 00:06:53.876891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876870    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4"} err="failed to get container status \"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4\": rpc error: code = NotFound desc = could not find container \"316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4\": container with ID starting with 316f9306d04650553787d1385bd5d040974e1244d8ed957b107565b8705117f4 not found: ID does not exist"
Nov 03 00:06:53.876957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876894    6000 scope.go:122] "RemoveContainer" containerID="6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"
Nov 03 00:06:53.877006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.876954    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:06:53.877139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877121    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019"} err="failed to get container status \"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019\": rpc error: code = NotFound desc = could not find container \"6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019\": container with ID starting with 6f24d06bd8c257f75e30bbda1f1667ae8697082c5db55b69c95160236d505019 not found: ID does not exist"
Nov 03 00:06:53.877202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877147    6000 scope.go:122] "RemoveContainer" containerID="65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"
Nov 03 00:06:53.877458 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877434    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a"} err="failed to get container status \"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a\": rpc error: code = NotFound desc = could not find container \"65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a\": container with ID starting with 65fd472894d518106aedc4cd7af3bdb63f0872a43c2014677cedc1fd04e51a2a not found: ID does not exist"
Nov 03 00:06:53.877531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877460    6000 scope.go:122] "RemoveContainer" containerID="2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"
Nov 03 00:06:53.877740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877719    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac"} err="failed to get container status \"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac\": rpc error: code = NotFound desc = could not find container \"2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac\": container with ID starting with 2b20996a6833f0a37f33409efc1503c58987ae0bcb4d8c015ab77345b9e847ac not found: ID does not exist"
Nov 03 00:06:53.877821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877741    6000 scope.go:122] "RemoveContainer" containerID="19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"
Nov 03 00:06:53.877982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877956    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547"} err="failed to get container status \"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547\": rpc error: code = NotFound desc = could not find container \"19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547\": container with ID starting with 19b35403cb27bdb5a13e638e80e5392d7454d5be08a3cbb98cf6953d6f3b6547 not found: ID does not exist"
Nov 03 00:06:53.878030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.877983    6000 scope.go:122] "RemoveContainer" containerID="432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6"
Nov 03 00:06:53.878189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.878166    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6"} err="failed to get container status \"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6\": rpc error: code = NotFound desc = could not find container \"432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6\": container with ID starting with 432a093d0f258b4caa607a7a404c47db43a966cae96bc26d801d4d0b9fcd4ac6 not found: ID does not exist"
Nov 03 00:06:53.880040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.880023    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:06:53.906456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906432    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:06:53.906801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906785    6000 cpu_manager.go:414] "containerMap: removing container" podUID="32ce8d43-d4b6-4b23-a235-624a0e30612d" containerName="console"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906804    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ce8d43-d4b6-4b23-a235-624a0e30612d" containerName="console"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906823    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="init-config-reloader"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906831    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="init-config-reloader"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906856    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-metric"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906864    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-metric"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906879    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="prom-label-proxy"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906884    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="prom-label-proxy"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906891    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-web"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906895    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-web"
Nov 03 00:06:53.906904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906905    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906911    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906924    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="alertmanager"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906930    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="alertmanager"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906951    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="config-reloader"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.906957    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="config-reloader"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907007    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-metric"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907015    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="prom-label-proxy"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907024    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy-web"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907035    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="kube-rbac-proxy"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907041    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="alertmanager"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907046    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="config-reloader"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907051    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ce8d43-d4b6-4b23-a235-624a0e30612d" containerName="console"
Nov 03 00:06:53.907348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.907057    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" containerName="init-config-reloader"
Nov 03 00:06:53.912505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.912482    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:53.926565 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926533    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 03 00:06:53.926667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926611    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 03 00:06:53.926667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926633    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 03 00:06:53.926842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926716    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 03 00:06:53.926842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926737    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tbmwn\""
Nov 03 00:06:53.926842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926828    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 03 00:06:53.926842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926839    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 03 00:06:53.927030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926909    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 03 00:06:53.927030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.926972    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 03 00:06:53.938585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.938565    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 03 00:06:53.939313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:53.939292    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:06:54.018433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018395    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dcd390-f475-4420-a193-e6f8204fc396-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018449    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018529    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-web-config\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018566    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018661    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dcd390-f475-4420-a193-e6f8204fc396-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018684    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018723    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dcd390-f475-4420-a193-e6f8204fc396-config-out\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018837 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018819    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018848    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018870    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dcd390-f475-4420-a193-e6f8204fc396-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.018990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.018953    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dcd390-f475-4420-a193-e6f8204fc396-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.019046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.019024    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.019108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.019085    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-768zj\" (UniqueName: \"kubernetes.io/projected/a6dcd390-f475-4420-a193-e6f8204fc396-kube-api-access-768zj\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120097    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dcd390-f475-4420-a193-e6f8204fc396-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120144    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dcd390-f475-4420-a193-e6f8204fc396-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120162    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120181    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-768zj\" (UniqueName: \"kubernetes.io/projected/a6dcd390-f475-4420-a193-e6f8204fc396-kube-api-access-768zj\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120206    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dcd390-f475-4420-a193-e6f8204fc396-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120229    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120242    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-web-config\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120279    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120330    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dcd390-f475-4420-a193-e6f8204fc396-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120362    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120409    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dcd390-f475-4420-a193-e6f8204fc396-config-out\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120446    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120473    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120598    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6dcd390-f475-4420-a193-e6f8204fc396-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.120768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.120626    6000 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 03 00:06:54.121452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.121025    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6dcd390-f475-4420-a193-e6f8204fc396-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.121452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.121082    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6dcd390-f475-4420-a193-e6f8204fc396-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.123747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.123722    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.123863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.123725    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.123863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.123815    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.124106 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.124085    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.124194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.124150    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6dcd390-f475-4420-a193-e6f8204fc396-config-out\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.124377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.124360    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.124509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.124491    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-web-config\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.124655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.124642    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6dcd390-f475-4420-a193-e6f8204fc396-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.125372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.125357    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6dcd390-f475-4420-a193-e6f8204fc396-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.140101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.140079    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-768zj\" (UniqueName: \"kubernetes.io/projected/a6dcd390-f475-4420-a193-e6f8204fc396-kube-api-access-768zj\") pod \"alertmanager-main-0\" (UID: \"a6dcd390-f475-4420-a193-e6f8204fc396\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.230118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.230094    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 03 00:06:54.552701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.552564    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 03 00:06:54.554950 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:06:54.554913    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6dcd390_f475_4420_a193_e6f8204fc396.slice/crio-5ad5749c70daa1e727c02bb94c159e744f407ba7e2165cffe1307e131a91d286 WatchSource:0}: Error finding container 5ad5749c70daa1e727c02bb94c159e744f407ba7e2165cffe1307e131a91d286: Status 404 returned error can't find the container with id 5ad5749c70daa1e727c02bb94c159e744f407ba7e2165cffe1307e131a91d286
Nov 03 00:06:54.564586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.564565    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8b2a49b-6679-40ce-91f0-91fcfd7c7118" path="/var/lib/kubelet/pods/e8b2a49b-6679-40ce-91f0-91fcfd7c7118/volumes"
Nov 03 00:06:54.830990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.830900    6000 generic.go:356] "Generic (PLEG): container finished" podID="a6dcd390-f475-4420-a193-e6f8204fc396" containerID="e69146b632a83a50e70bc7893cac4b72d48dfca536a5920a54a3d835484e7068" exitCode=0
Nov 03 00:06:54.831129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.830984    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerDied","Data":"e69146b632a83a50e70bc7893cac4b72d48dfca536a5920a54a3d835484e7068"}
Nov 03 00:06:54.831129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:54.831049    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerStarted","Data":"5ad5749c70daa1e727c02bb94c159e744f407ba7e2165cffe1307e131a91d286"}
Nov 03 00:06:55.838201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:55.838168    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerStarted","Data":"1f9ca7f28531e7c1be95dbcb671167bca50fddee1d7b20e995492d10300cfb39"}
Nov 03 00:06:55.838587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:55.838276    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerStarted","Data":"389972f348c868f254bab0dfb969055677fec2a3a47899657c1429daa995959b"}
Nov 03 00:06:55.838587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:55.838303    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerStarted","Data":"77feccb324d911c7457b5a68a399293cbc06aa7cdd9ef4a419277cf0f34e605b"}
Nov 03 00:06:55.838587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:55.838328    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerStarted","Data":"71c16b4b0d93069cc2e13b2c1b6eb89b30e568ea3ebd1475a06c7ab982185cd0"}
Nov 03 00:06:55.838587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:55.838351    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerStarted","Data":"0f8462a2b61e24866f61bce0827471f0924e19e1177c39cbb2e139c8aa937338"}
Nov 03 00:06:55.838587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:55.838373    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6dcd390-f475-4420-a193-e6f8204fc396","Type":"ContainerStarted","Data":"b1cf54636b4749f87cb06c8b506a940cea07c75bdffd0ca1d68615b5d62184a9"}
Nov 03 00:06:55.904748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:06:55.904699    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.892234189 podStartE2EDuration="2.904684428s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.450239ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:06:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:06:55.892824475 +0000 UTC m=+53.448176498" watchObservedRunningTime="2026-11-03 00:06:55.904684428 +0000 UTC m=+53.460036454"
Nov 03 00:07:10.041441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.041411    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-664c8f59ff-w89r9"]
Nov 03 00:07:10.045150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.045123    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.057128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.057097    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-serving-cert\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.057355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.057333    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-service-ca\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.057493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.057477    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-oauth-config\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.057594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.057581    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-tmp\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.057698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.057684    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-trusted-ca-bundle\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.057819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.057806    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-config\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.057952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.057939    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-oauth-serving-cert\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.058055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.058041    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2jr4\" (UniqueName: \"kubernetes.io/projected/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-kube-api-access-p2jr4\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.071983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.071957    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664c8f59ff-w89r9"]
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160693    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-serving-cert\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160754    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-service-ca\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160796    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-oauth-config\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160821    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-tmp\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160851    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-trusted-ca-bundle\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160882    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-config\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160939    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-oauth-serving-cert\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.160967    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2jr4\" (UniqueName: \"kubernetes.io/projected/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-kube-api-access-p2jr4\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.162625    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-tmp\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.163291    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-service-ca\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.164007    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-oauth-serving-cert\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.166279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.164992    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-trusted-ca-bundle\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.169601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.169579    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-config\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.171291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.171227    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-oauth-config\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.171423 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.171407    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-console-serving-cert\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.180134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.180105    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2jr4\" (UniqueName: \"kubernetes.io/projected/416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6-kube-api-access-p2jr4\") pod \"console-664c8f59ff-w89r9\" (UID: \"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6\") " pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.357422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.357393    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:10.654354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.654069    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-664c8f59ff-w89r9"]
Nov 03 00:07:10.656381 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:07:10.656353    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod416f8d8f_5e16_49aa_b2a5_6140c2bc4ef6.slice/crio-632017aaf34b066c8905e60fb2569999530566f6d22c19b43ae46f864879dfb1 WatchSource:0}: Error finding container 632017aaf34b066c8905e60fb2569999530566f6d22c19b43ae46f864879dfb1: Status 404 returned error can't find the container with id 632017aaf34b066c8905e60fb2569999530566f6d22c19b43ae46f864879dfb1
Nov 03 00:07:10.897407 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.897375    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664c8f59ff-w89r9" event={"ID":"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6","Type":"ContainerStarted","Data":"7dcfc932730b6ff642cda79d23a81cf683c2411ff8223b2e390f90544b319485"}
Nov 03 00:07:10.897596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.897422    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-664c8f59ff-w89r9" event={"ID":"416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6","Type":"ContainerStarted","Data":"632017aaf34b066c8905e60fb2569999530566f6d22c19b43ae46f864879dfb1"}
Nov 03 00:07:10.935583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:10.935502    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-664c8f59ff-w89r9" podStartSLOduration=0.935489286 podStartE2EDuration="935.489286ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:07:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:07:10.934438822 +0000 UTC m=+68.489790847" watchObservedRunningTime="2026-11-03 00:07:10.935489286 +0000 UTC m=+68.490841309"
Nov 03 00:07:19.893229 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:07:19.893183    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:07:20.358213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:20.358187    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:20.358413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:20.358276    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:20.363037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:20.363018    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:20.936568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:20.936540    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-664c8f59ff-w89r9"
Nov 03 00:07:21.003208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:21.003178    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fcfc8dbfc-dspqk"]
Nov 03 00:07:46.027904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.027798    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5fcfc8dbfc-dspqk" podUID="1d1799f5-bac5-4c4c-b399-6f1a5130e140" containerName="console" containerID="cri-o://ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b" gracePeriod=15
Nov 03 00:07:46.271766 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.271745    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:07:46.330119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330051    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330086    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330115    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330135    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330320    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\" (UniqueName: \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330368    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330554 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330412    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330554 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330464    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\") pod \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\" (UID: \"1d1799f5-bac5-4c4c-b399-6f1a5130e140\") "
Nov 03 00:07:46.330554 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330520    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:07:46.330697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330544    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:07:46.330697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330578    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:07:46.330697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330589    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:07:46.330833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330710    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:07:46.330948 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330930    6000 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:46.331012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330953    6000 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-oauth-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:46.331012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330967    6000 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-trusted-ca-bundle\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:46.331012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330981    6000 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1d1799f5-bac5-4c4c-b399-6f1a5130e140-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:46.331012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.330995    6000 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1d1799f5-bac5-4c4c-b399-6f1a5130e140-tmp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:46.332688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.332664    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "kube-api-access-bxrnc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:07:46.332800 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.332685    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:07:46.332800 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.332762    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert" pod "1d1799f5-bac5-4c4c-b399-6f1a5130e140" (UID: "1d1799f5-bac5-4c4c-b399-6f1a5130e140"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:07:46.432291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.432233    6000 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-serving-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:46.432291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.432290    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxrnc\" (UniqueName: \"kubernetes.io/projected/1d1799f5-bac5-4c4c-b399-6f1a5130e140-kube-api-access-bxrnc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:46.432291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:46.432300    6000 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1d1799f5-bac5-4c4c-b399-6f1a5130e140-console-oauth-config\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:07:47.027282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.027237    6000 generic.go:356] "Generic (PLEG): container finished" podID="1d1799f5-bac5-4c4c-b399-6f1a5130e140" containerID="ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b" exitCode=0
Nov 03 00:07:47.027456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.027301    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5fcfc8dbfc-dspqk"
Nov 03 00:07:47.027456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.027322    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fcfc8dbfc-dspqk" event={"ID":"1d1799f5-bac5-4c4c-b399-6f1a5130e140","Type":"ContainerDied","Data":"ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b"}
Nov 03 00:07:47.027456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.027375    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5fcfc8dbfc-dspqk" event={"ID":"1d1799f5-bac5-4c4c-b399-6f1a5130e140","Type":"ContainerDied","Data":"513f34b7127724cbf28ae9b850e22058500c84f260a429dc48f44205cebced74"}
Nov 03 00:07:47.027456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.027394    6000 scope.go:122] "RemoveContainer" containerID="ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b"
Nov 03 00:07:47.037857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.037835    6000 scope.go:122] "RemoveContainer" containerID="ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b"
Nov 03 00:07:47.038170 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:07:47.038146    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b\": container with ID starting with ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b not found: ID does not exist" containerID="ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b"
Nov 03 00:07:47.038218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.038173    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b"} err="failed to get container status \"ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b\": rpc error: code = NotFound desc = could not find container \"ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b\": container with ID starting with ec4cd5b590b70e23e64632158eed0089b7d91007936392c1dd74a911ce11435b not found: ID does not exist"
Nov 03 00:07:47.076235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.076208    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5fcfc8dbfc-dspqk"]
Nov 03 00:07:47.078775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:47.078756    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5fcfc8dbfc-dspqk"]
Nov 03 00:07:48.564138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:07:48.564103    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d1799f5-bac5-4c4c-b399-6f1a5130e140" path="/var/lib/kubelet/pods/1d1799f5-bac5-4c4c-b399-6f1a5130e140/volumes"
Nov 03 00:07:49.894411 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:07:49.894379    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:08:19.894783 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:08:19.894746    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:08:49.893611 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:08:49.893572    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:08:50.207922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.207830    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"]
Nov 03 00:08:50.208247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.208229    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1d1799f5-bac5-4c4c-b399-6f1a5130e140" containerName="console"
Nov 03 00:08:50.208247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.208250    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d1799f5-bac5-4c4c-b399-6f1a5130e140" containerName="console"
Nov 03 00:08:50.208422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.208365    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d1799f5-bac5-4c4c-b399-6f1a5130e140" containerName="console"
Nov 03 00:08:50.210426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.210404    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.226540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.226519    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 03 00:08:50.228809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.228793    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 03 00:08:50.228892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.228878    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 03 00:08:50.238459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.238436    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 00:08:50.238576 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.238469    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 03 00:08:50.238576 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.238480    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wclrs\""
Nov 03 00:08:50.240587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.240563    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 03 00:08:50.241375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.241356    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"]
Nov 03 00:08:50.335324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.335289    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49ztv\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-kube-api-access-49ztv\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.335324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.335323    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9810b119-3076-49fd-9c52-140de7ec9b1f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.335543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.335341    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.335543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.335358    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.335543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.335479    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.335704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.335540    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.335704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.335631    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.435993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.435961    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.435993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.435998    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.436212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.436121    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49ztv\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-kube-api-access-49ztv\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.436212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.436155    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9810b119-3076-49fd-9c52-140de7ec9b1f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.436212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.436172    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.436212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.436189    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.436212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.436210    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.436651 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.436617    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.438628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.438598    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9810b119-3076-49fd-9c52-140de7ec9b1f-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.438778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.438761    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.438849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.438817    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.438849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.438826    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.456140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.456114    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.456319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.456304    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49ztv\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-kube-api-access-49ztv\") pod \"istiod-openshift-gateway-7876bfcb74-8n6rv\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.521387 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.521334    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:50.827548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.827520    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"]
Nov 03 00:08:50.829658 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:08:50.829630    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9810b119_3076_49fd_9c52_140de7ec9b1f.slice/crio-4d7c663811722306a2d7fd6ff7758fc3b9b7e29d03fe77b59d312584e7e3db51 WatchSource:0}: Error finding container 4d7c663811722306a2d7fd6ff7758fc3b9b7e29d03fe77b59d312584e7e3db51: Status 404 returned error can't find the container with id 4d7c663811722306a2d7fd6ff7758fc3b9b7e29d03fe77b59d312584e7e3db51
Nov 03 00:08:50.831622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:50.831605    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:08:51.251809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:51.251764    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" event={"ID":"9810b119-3076-49fd-9c52-140de7ec9b1f","Type":"ContainerStarted","Data":"4d7c663811722306a2d7fd6ff7758fc3b9b7e29d03fe77b59d312584e7e3db51"}
Nov 03 00:08:54.589560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:54.589533    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:08:54.589826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:54.589602    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:08:55.269263 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:55.269229    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" event={"ID":"9810b119-3076-49fd-9c52-140de7ec9b1f","Type":"ContainerStarted","Data":"01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0"}
Nov 03 00:08:55.269548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:55.269458    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:08:55.271099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:55.271070    6000 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8n6rv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:08:55.271224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:55.271129    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" podUID="9810b119-3076-49fd-9c52-140de7ec9b1f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:08:55.344595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:55.344509    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" podStartSLOduration=1.587596441 podStartE2EDuration="5.344491657s" totalImagesPullingTime="3.756895216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:08:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:08:55.32841863 +0000 UTC m=+172.883770655" watchObservedRunningTime="2026-11-03 00:08:55.344491657 +0000 UTC m=+172.899843681"
Nov 03 00:08:56.273759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:08:56.273734    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:09:19.893869 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:09:19.893792    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:09:49.893805 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:09:49.893768    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:09:56.241113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.241077    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"]
Nov 03 00:09:56.244460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.244444    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.291922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.291884    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"]
Nov 03 00:09:56.341230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.341195    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 03 00:09:56.341455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.341382    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 03 00:09:56.341455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.341389    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 03 00:09:56.341455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.341399    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kvgpd\""
Nov 03 00:09:56.341455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.341432    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 03 00:09:56.341455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.341443    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 03 00:09:56.384995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.384958    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.385178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.384999    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ca00-0922-44be-b24b-0960fb3d449d-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.385178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.385032    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s6fq\" (UniqueName: \"kubernetes.io/projected/3ce9ca00-0922-44be-b24b-0960fb3d449d-kube-api-access-9s6fq\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.385178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.385115    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ca00-0922-44be-b24b-0960fb3d449d-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.385178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.385138    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.385178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.385178    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.486596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.486567    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.486796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.486623    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.486796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.486663    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.486796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.486692    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ca00-0922-44be-b24b-0960fb3d449d-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.486796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.486709    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s6fq\" (UniqueName: \"kubernetes.io/projected/3ce9ca00-0922-44be-b24b-0960fb3d449d-kube-api-access-9s6fq\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.486796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.486737    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ca00-0922-44be-b24b-0960fb3d449d-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.487226 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.487195    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ca00-0922-44be-b24b-0960fb3d449d-manifests\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.489360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.489338    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-apiservice-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.489507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.489357    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3ce9ca00-0922-44be-b24b-0960fb3d449d-tmp\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.489507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.489473    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-metrics-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.489623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.489568    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/3ce9ca00-0922-44be-b24b-0960fb3d449d-webhook-cert\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.505041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.504982    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s6fq\" (UniqueName: \"kubernetes.io/projected/3ce9ca00-0922-44be-b24b-0960fb3d449d-kube-api-access-9s6fq\") pod \"opendatahub-operator-controller-manager-566cf6f9f5-zczkf\" (UID: \"3ce9ca00-0922-44be-b24b-0960fb3d449d\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.559183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.559153    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:09:56.903403 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:56.903372    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"]
Nov 03 00:09:56.916161 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:09:56.916134    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ce9ca00_0922_44be_b24b_0960fb3d449d.slice/crio-2aa2b027fa23adf10040174cec5c3072f3ffb05a0973247d5a4f904b45dd9cfc WatchSource:0}: Error finding container 2aa2b027fa23adf10040174cec5c3072f3ffb05a0973247d5a4f904b45dd9cfc: Status 404 returned error can't find the container with id 2aa2b027fa23adf10040174cec5c3072f3ffb05a0973247d5a4f904b45dd9cfc
Nov 03 00:09:57.510292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:09:57.510224    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf" event={"ID":"3ce9ca00-0922-44be-b24b-0960fb3d449d","Type":"ContainerStarted","Data":"2aa2b027fa23adf10040174cec5c3072f3ffb05a0973247d5a4f904b45dd9cfc"}
Nov 03 00:10:00.524443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:00.524410    6000 generic.go:356] "Generic (PLEG): container finished" podID="3ce9ca00-0922-44be-b24b-0960fb3d449d" containerID="cfc9ba30f37976b51912c47d54ae13ceaac3605ac75d659c24ca55d1cc234032" exitCode=0
Nov 03 00:10:00.524824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:00.524469    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf" event={"ID":"3ce9ca00-0922-44be-b24b-0960fb3d449d","Type":"ContainerDied","Data":"cfc9ba30f37976b51912c47d54ae13ceaac3605ac75d659c24ca55d1cc234032"}
Nov 03 00:10:01.531337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:01.531298    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf" event={"ID":"3ce9ca00-0922-44be-b24b-0960fb3d449d","Type":"ContainerStarted","Data":"6de1f12e5b16bd33309a1ef39c7921ce751542f2c46167c29f15fb70dca2b78b"}
Nov 03 00:10:01.531755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:01.531469    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:10:01.608311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:01.608235    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf" podStartSLOduration=1.845335753 podStartE2EDuration="5.608220668s" totalImagesPullingTime="3.293553156s" totalInitContainerRuntime="469.331759ms" isStatefulPod=false podCreationTimestamp="2026-11-03 00:09:56 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:01.59615475 +0000 UTC m=+239.151506773" watchObservedRunningTime="2026-11-03 00:10:01.608220668 +0000 UTC m=+239.163572692"
Nov 03 00:10:12.539640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:12.539608    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-566cf6f9f5-zczkf"
Nov 03 00:10:19.894272 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:10:19.894221    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:10:28.804090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.804054    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"]
Nov 03 00:10:28.809111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.809092    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:28.842817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.842786    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 03 00:10:28.842994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.842892    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 03 00:10:28.843062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.843000    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-gr7kg\""
Nov 03 00:10:28.843869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.843838    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"]
Nov 03 00:10:28.878700 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.878670    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs84c\" (UniqueName: \"kubernetes.io/projected/efd0e509-7023-4876-b16f-410990d2c1f1-kube-api-access-cs84c\") pod \"cert-manager-webhook-5dbbbf75cc-6l596\" (UID: \"efd0e509-7023-4876-b16f-410990d2c1f1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:28.878841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.878721    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efd0e509-7023-4876-b16f-410990d2c1f1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6l596\" (UID: \"efd0e509-7023-4876-b16f-410990d2c1f1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:28.980281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.980220    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs84c\" (UniqueName: \"kubernetes.io/projected/efd0e509-7023-4876-b16f-410990d2c1f1-kube-api-access-cs84c\") pod \"cert-manager-webhook-5dbbbf75cc-6l596\" (UID: \"efd0e509-7023-4876-b16f-410990d2c1f1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:28.980461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:28.980320    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efd0e509-7023-4876-b16f-410990d2c1f1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6l596\" (UID: \"efd0e509-7023-4876-b16f-410990d2c1f1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:29.007851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:29.007821    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/efd0e509-7023-4876-b16f-410990d2c1f1-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-6l596\" (UID: \"efd0e509-7023-4876-b16f-410990d2c1f1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:29.007993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:29.007821    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs84c\" (UniqueName: \"kubernetes.io/projected/efd0e509-7023-4876-b16f-410990d2c1f1-kube-api-access-cs84c\") pod \"cert-manager-webhook-5dbbbf75cc-6l596\" (UID: \"efd0e509-7023-4876-b16f-410990d2c1f1\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:29.133326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:29.133294    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:29.439911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:29.439884    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"]
Nov 03 00:10:29.441200 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:10:29.441172    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podefd0e509_7023_4876_b16f_410990d2c1f1.slice/crio-8751ce650509a11ef698d02eb80350c48948e8238919c82ff9422efbbf8033a4 WatchSource:0}: Error finding container 8751ce650509a11ef698d02eb80350c48948e8238919c82ff9422efbbf8033a4: Status 404 returned error can't find the container with id 8751ce650509a11ef698d02eb80350c48948e8238919c82ff9422efbbf8033a4
Nov 03 00:10:29.645967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:29.645934    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596" event={"ID":"efd0e509-7023-4876-b16f-410990d2c1f1","Type":"ContainerStarted","Data":"8751ce650509a11ef698d02eb80350c48948e8238919c82ff9422efbbf8033a4"}
Nov 03 00:10:32.291284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.291221    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"]
Nov 03 00:10:32.294525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.294502    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.313957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.313931    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 03 00:10:32.321711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.321655    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cwbt5\""
Nov 03 00:10:32.321946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.321928    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 03 00:10:32.322630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.322602    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"]
Nov 03 00:10:32.412578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.412542    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9847f\" (UniqueName: \"kubernetes.io/projected/7be7a5b4-d36a-48a6-afc8-40741783c4fa-kube-api-access-9847f\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.412755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.412640    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be7a5b4-d36a-48a6-afc8-40741783c4fa-tls-certs\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.412836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.412752    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7be7a5b4-d36a-48a6-afc8-40741783c4fa-tmp\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.513433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.513396    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9847f\" (UniqueName: \"kubernetes.io/projected/7be7a5b4-d36a-48a6-afc8-40741783c4fa-kube-api-access-9847f\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.513638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.513491    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be7a5b4-d36a-48a6-afc8-40741783c4fa-tls-certs\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.513638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.513542    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7be7a5b4-d36a-48a6-afc8-40741783c4fa-tmp\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.516574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.516541    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7be7a5b4-d36a-48a6-afc8-40741783c4fa-tmp\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.517418 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.517390    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7be7a5b4-d36a-48a6-afc8-40741783c4fa-tls-certs\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.535978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.535953    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9847f\" (UniqueName: \"kubernetes.io/projected/7be7a5b4-d36a-48a6-afc8-40741783c4fa-kube-api-access-9847f\") pod \"kube-auth-proxy-66c6b6454f-hlqrl\" (UID: \"7be7a5b4-d36a-48a6-afc8-40741783c4fa\") " pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.609513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.609476    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"
Nov 03 00:10:32.921067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:32.921038    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl"]
Nov 03 00:10:32.922728 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:10:32.922700    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be7a5b4_d36a_48a6_afc8_40741783c4fa.slice/crio-255e4e2622c7cc649bc45f414c510813e63431792ecb0d8d4f97d4a7c6301511 WatchSource:0}: Error finding container 255e4e2622c7cc649bc45f414c510813e63431792ecb0d8d4f97d4a7c6301511: Status 404 returned error can't find the container with id 255e4e2622c7cc649bc45f414c510813e63431792ecb0d8d4f97d4a7c6301511
Nov 03 00:10:33.671562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:33.671441    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596" event={"ID":"efd0e509-7023-4876-b16f-410990d2c1f1","Type":"ContainerStarted","Data":"8cd5526cd2339dec6e538c74703fc5df649e499a9edd204907aae582593211a7"}
Nov 03 00:10:33.671562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:33.671529    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:33.673103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:33.673068    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl" event={"ID":"7be7a5b4-d36a-48a6-afc8-40741783c4fa","Type":"ContainerStarted","Data":"255e4e2622c7cc649bc45f414c510813e63431792ecb0d8d4f97d4a7c6301511"}
Nov 03 00:10:34.582772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.582714    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596" podStartSLOduration=3.011708774 podStartE2EDuration="6.582698221s" totalImagesPullingTime="3.570989447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:33.728847152 +0000 UTC m=+271.284199178" watchObservedRunningTime="2026-11-03 00:10:34.582698221 +0000 UTC m=+272.138050239"
Nov 03 00:10:34.582971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.582928    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:10:34.589824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.589796    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:34.611198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.606109    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:10:34.653386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.653352    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 03 00:10:34.653547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.653412    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 03 00:10:34.653547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.653434    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 03 00:10:34.666691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.666646    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cl6jh\""
Nov 03 00:10:34.683944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.683915    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:10:34.733438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.733332    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f109704e-d9c1-41e3-84c9-5a0aa3eac6be-config-file\") pod \"limitador-limitador-7c9b87d574-kxq2l\" (UID: \"f109704e-d9c1-41e3-84c9-5a0aa3eac6be\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:34.733438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.733405    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mlft\" (UniqueName: \"kubernetes.io/projected/f109704e-d9c1-41e3-84c9-5a0aa3eac6be-kube-api-access-5mlft\") pod \"limitador-limitador-7c9b87d574-kxq2l\" (UID: \"f109704e-d9c1-41e3-84c9-5a0aa3eac6be\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:34.835313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.835216    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f109704e-d9c1-41e3-84c9-5a0aa3eac6be-config-file\") pod \"limitador-limitador-7c9b87d574-kxq2l\" (UID: \"f109704e-d9c1-41e3-84c9-5a0aa3eac6be\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:34.835480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.835321    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mlft\" (UniqueName: \"kubernetes.io/projected/f109704e-d9c1-41e3-84c9-5a0aa3eac6be-kube-api-access-5mlft\") pod \"limitador-limitador-7c9b87d574-kxq2l\" (UID: \"f109704e-d9c1-41e3-84c9-5a0aa3eac6be\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:34.837294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.835964    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/f109704e-d9c1-41e3-84c9-5a0aa3eac6be-config-file\") pod \"limitador-limitador-7c9b87d574-kxq2l\" (UID: \"f109704e-d9c1-41e3-84c9-5a0aa3eac6be\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:34.855783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.855757    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mlft\" (UniqueName: \"kubernetes.io/projected/f109704e-d9c1-41e3-84c9-5a0aa3eac6be-kube-api-access-5mlft\") pod \"limitador-limitador-7c9b87d574-kxq2l\" (UID: \"f109704e-d9c1-41e3-84c9-5a0aa3eac6be\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:34.913655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:34.913621    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:35.223617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:35.223585    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:10:35.226710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:10:35.226677    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf109704e_d9c1_41e3_84c9_5a0aa3eac6be.slice/crio-36e28ad5b106c71547d6dc34af8b7fcce53b9e928a7c75d76181939c825b8c04 WatchSource:0}: Error finding container 36e28ad5b106c71547d6dc34af8b7fcce53b9e928a7c75d76181939c825b8c04: Status 404 returned error can't find the container with id 36e28ad5b106c71547d6dc34af8b7fcce53b9e928a7c75d76181939c825b8c04
Nov 03 00:10:35.683581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:35.683544    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl" event={"ID":"7be7a5b4-d36a-48a6-afc8-40741783c4fa","Type":"ContainerStarted","Data":"b9abdc50b1ce5f56c594268441667506da38e1508da7a8954ae78fe8f97be7fe"}
Nov 03 00:10:35.684802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:35.684774    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l" event={"ID":"f109704e-d9c1-41e3-84c9-5a0aa3eac6be","Type":"ContainerStarted","Data":"36e28ad5b106c71547d6dc34af8b7fcce53b9e928a7c75d76181939c825b8c04"}
Nov 03 00:10:35.721735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:35.721676    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-66c6b6454f-hlqrl" podStartSLOduration=1.326686836 podStartE2EDuration="3.721658113s" totalImagesPullingTime="2.394971277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:35.719561248 +0000 UTC m=+273.274913287" watchObservedRunningTime="2026-11-03 00:10:35.721658113 +0000 UTC m=+273.277010138"
Nov 03 00:10:37.695342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:37.695299    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l" event={"ID":"f109704e-d9c1-41e3-84c9-5a0aa3eac6be","Type":"ContainerStarted","Data":"72cd8be3a4a0acc9d8fd6ffabdadd8fb1fa9fa267576f762ba75db5746c7f525"}
Nov 03 00:10:37.695792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:37.695370    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:37.735611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:37.735517    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l" podStartSLOduration=1.438756566 podStartE2EDuration="3.735497171s" totalImagesPullingTime="2.296740605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:10:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:10:37.735087765 +0000 UTC m=+275.290439790" watchObservedRunningTime="2026-11-03 00:10:37.735497171 +0000 UTC m=+275.290849206"
Nov 03 00:10:39.680500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:39.680467    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-6l596"
Nov 03 00:10:48.700155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:10:48.700070    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"
Nov 03 00:10:49.894713 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:10:49.894665    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:11:19.893551 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:11:19.893501    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:11:23.853445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.853407    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-d9d5b6dc8-2n7g9"]
Nov 03 00:11:23.856278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.856240    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:23.872386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.872366    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 03 00:11:23.880530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.880510    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 00:11:23.880657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.880554    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 03 00:11:23.880657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.880621    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 00:11:23.883631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.883585    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sfqns\""
Nov 03 00:11:23.883764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.883654    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 00:11:23.884662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.884642    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d9d5b6dc8-2n7g9"]
Nov 03 00:11:23.953353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.953319    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-metrics-certs\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:23.953522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.953382    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckgk2\" (UniqueName: \"kubernetes.io/projected/13edb37b-2005-4a92-9833-8850cdcb2157-kube-api-access-ckgk2\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:23.953589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:23.953561    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-webhook-cert\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.054621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.054577    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckgk2\" (UniqueName: \"kubernetes.io/projected/13edb37b-2005-4a92-9833-8850cdcb2157-kube-api-access-ckgk2\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.054805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.054634    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-webhook-cert\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.054805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.054694    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-metrics-certs\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.057382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.057358    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-metrics-certs\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.057512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.057386    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-webhook-cert\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.072865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.072832    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckgk2\" (UniqueName: \"kubernetes.io/projected/13edb37b-2005-4a92-9833-8850cdcb2157-kube-api-access-ckgk2\") pod \"maas-controller-d9d5b6dc8-2n7g9\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") " pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.168174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.168094    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:24.307038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.307006    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-9crlh"]
Nov 03 00:11:24.309877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.309836    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.330866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.330839    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-9crlh"]
Nov 03 00:11:24.458275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.458155    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-webhook-cert\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.458475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.458302    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-metrics-certs\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.458475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.458349    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56p25\" (UniqueName: \"kubernetes.io/projected/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-kube-api-access-56p25\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.465833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.465803    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-d9d5b6dc8-2n7g9"]
Nov 03 00:11:24.468397 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:11:24.468372    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13edb37b_2005_4a92_9833_8850cdcb2157.slice/crio-454e3847cb44b13e66bcfac8c2d32345fdc5f345f021a17084ef2064ab7bc30d WatchSource:0}: Error finding container 454e3847cb44b13e66bcfac8c2d32345fdc5f345f021a17084ef2064ab7bc30d: Status 404 returned error can't find the container with id 454e3847cb44b13e66bcfac8c2d32345fdc5f345f021a17084ef2064ab7bc30d
Nov 03 00:11:24.559659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.559629    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-metrics-certs\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.559863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.559694    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56p25\" (UniqueName: \"kubernetes.io/projected/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-kube-api-access-56p25\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.559863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.559757    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-webhook-cert\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.562518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.562490    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-metrics-certs\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.562644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.562490    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-webhook-cert\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.577718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.577694    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56p25\" (UniqueName: \"kubernetes.io/projected/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-kube-api-access-56p25\") pod \"maas-controller-bc8c7d7fb-9crlh\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") " pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.622241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.622216    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:24.886316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.886286    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9" event={"ID":"13edb37b-2005-4a92-9833-8850cdcb2157","Type":"ContainerStarted","Data":"454e3847cb44b13e66bcfac8c2d32345fdc5f345f021a17084ef2064ab7bc30d"}
Nov 03 00:11:24.914127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:24.914106    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-9crlh"]
Nov 03 00:11:24.915727 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:11:24.915699    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65984e3f_0d1a_4d8c_ac3a_ca76eda948a5.slice/crio-d1cddab9f0a0cba5ae1dd784f40f7a5965484e350d884c8b2636813c68108dd4 WatchSource:0}: Error finding container d1cddab9f0a0cba5ae1dd784f40f7a5965484e350d884c8b2636813c68108dd4: Status 404 returned error can't find the container with id d1cddab9f0a0cba5ae1dd784f40f7a5965484e350d884c8b2636813c68108dd4
Nov 03 00:11:25.894295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:25.894238    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" event={"ID":"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5","Type":"ContainerStarted","Data":"d1cddab9f0a0cba5ae1dd784f40f7a5965484e350d884c8b2636813c68108dd4"}
Nov 03 00:11:27.905866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:27.905835    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9" event={"ID":"13edb37b-2005-4a92-9833-8850cdcb2157","Type":"ContainerStarted","Data":"9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa"}
Nov 03 00:11:27.906352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:27.905954    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:27.907521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:27.907498    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" event={"ID":"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5","Type":"ContainerStarted","Data":"4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec"}
Nov 03 00:11:27.907647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:27.907546    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:27.950170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:27.950108    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9" podStartSLOduration=2.281519921 podStartE2EDuration="4.950092733s" totalImagesPullingTime="2.668572812s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:27.950092676 +0000 UTC m=+325.505444694" watchObservedRunningTime="2026-11-03 00:11:27.950092733 +0000 UTC m=+325.505444757"
Nov 03 00:11:27.989090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:27.989034    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" podStartSLOduration=1.390931357 podStartE2EDuration="3.989019639s" totalImagesPullingTime="2.598088282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:27.987814741 +0000 UTC m=+325.543166778" watchObservedRunningTime="2026-11-03 00:11:27.989019639 +0000 UTC m=+325.544371663"
Nov 03 00:11:28.948658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:28.948624    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"]
Nov 03 00:11:28.950981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:28.950963    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:28.982633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:28.982611    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Nov 03 00:11:28.982750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:28.982653    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Nov 03 00:11:28.982750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:28.982684    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Nov 03 00:11:28.982818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:28.982748    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-62xr4\""
Nov 03 00:11:28.986793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:28.986772    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"]
Nov 03 00:11:29.108729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.108690    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f731881-a91d-48e5-b881-8bee6561c7fb-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.108889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.108755    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6f731881-a91d-48e5-b881-8bee6561c7fb-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.108889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.108854    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f731881-a91d-48e5-b881-8bee6561c7fb-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.108969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.108894    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcbrq\" (UniqueName: \"kubernetes.io/projected/6f731881-a91d-48e5-b881-8bee6561c7fb-kube-api-access-mcbrq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.209698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.209612    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f731881-a91d-48e5-b881-8bee6561c7fb-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.209698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.209680    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6f731881-a91d-48e5-b881-8bee6561c7fb-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.209910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.209721    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f731881-a91d-48e5-b881-8bee6561c7fb-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.209910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.209747    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcbrq\" (UniqueName: \"kubernetes.io/projected/6f731881-a91d-48e5-b881-8bee6561c7fb-kube-api-access-mcbrq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.210335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.210309    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/6f731881-a91d-48e5-b881-8bee6561c7fb-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.210581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.210561    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/6f731881-a91d-48e5-b881-8bee6561c7fb-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.212338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.212310    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/6f731881-a91d-48e5-b881-8bee6561c7fb-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.228920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.228901    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcbrq\" (UniqueName: \"kubernetes.io/projected/6f731881-a91d-48e5-b881-8bee6561c7fb-kube-api-access-mcbrq\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-hm874\" (UID: \"6f731881-a91d-48e5-b881-8bee6561c7fb\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.262903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.262879    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:29.585605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.585580    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"]
Nov 03 00:11:29.587230 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:11:29.587199    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f731881_a91d_48e5_b881_8bee6561c7fb.slice/crio-3a7e38a3e9ba2b830bb8cc43a7a7a2c1ae923552fed6d71aaffbaf094d65a8ff WatchSource:0}: Error finding container 3a7e38a3e9ba2b830bb8cc43a7a7a2c1ae923552fed6d71aaffbaf094d65a8ff: Status 404 returned error can't find the container with id 3a7e38a3e9ba2b830bb8cc43a7a7a2c1ae923552fed6d71aaffbaf094d65a8ff
Nov 03 00:11:29.919656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:29.919621    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874" event={"ID":"6f731881-a91d-48e5-b881-8bee6561c7fb","Type":"ContainerStarted","Data":"3a7e38a3e9ba2b830bb8cc43a7a7a2c1ae923552fed6d71aaffbaf094d65a8ff"}
Nov 03 00:11:35.790914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.790871    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"]
Nov 03 00:11:35.793626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.793593    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.820559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.820526    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"]
Nov 03 00:11:35.877380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.877341    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.877566 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.877406    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.877566 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.877508    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tz8p\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-kube-api-access-9tz8p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.877670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.877590    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.877670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.877627    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.877670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.877651    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.877803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.877763    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/794f1a8c-6a7c-4289-8907-b9450faa6d04-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.978510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.978472    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.978510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.978514    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.978731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.978530    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.978731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.978557    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/794f1a8c-6a7c-4289-8907-b9450faa6d04-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.978731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.978616    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.978731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.978661    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.978731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.978687    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tz8p\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-kube-api-access-9tz8p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.979649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.979620    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.981217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.981187    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.981329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.981243    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.981329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.981240    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/794f1a8c-6a7c-4289-8907-b9450faa6d04-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.981408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.981327    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.997432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.997407    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tz8p\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-kube-api-access-9tz8p\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:35.997602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:35.997581    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2wkww\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:36.106543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:36.106511    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:36.419914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:36.419776    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"]
Nov 03 00:11:36.424809 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:11:36.424537    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794f1a8c_6a7c_4289_8907_b9450faa6d04.slice/crio-24128938d24d7a9fc3e9aecb63bd4594752579425476320175afe377bc191d13 WatchSource:0}: Error finding container 24128938d24d7a9fc3e9aecb63bd4594752579425476320175afe377bc191d13: Status 404 returned error can't find the container with id 24128938d24d7a9fc3e9aecb63bd4594752579425476320175afe377bc191d13
Nov 03 00:11:36.427011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:36.426975    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:11:36.427128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:36.427075    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:11:36.966022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:36.965984    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww" event={"ID":"794f1a8c-6a7c-4289-8907-b9450faa6d04","Type":"ContainerStarted","Data":"82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11"}
Nov 03 00:11:36.966513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:36.966044    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww" event={"ID":"794f1a8c-6a7c-4289-8907-b9450faa6d04","Type":"ContainerStarted","Data":"24128938d24d7a9fc3e9aecb63bd4594752579425476320175afe377bc191d13"}
Nov 03 00:11:36.966513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:36.966308    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:37.020766 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:37.020712    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww" podStartSLOduration=2.020698197 podStartE2EDuration="2.020698197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:37.019168617 +0000 UTC m=+334.574520640" watchObservedRunningTime="2026-11-03 00:11:37.020698197 +0000 UTC m=+334.576050215"
Nov 03 00:11:37.973498 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:37.973472    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 00:11:38.054203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.054176    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"]
Nov 03 00:11:38.054434 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.054413    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" podUID="9810b119-3076-49fd-9c52-140de7ec9b1f" containerName="discovery" containerID="cri-o://01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0" gracePeriod=30
Nov 03 00:11:38.308336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.308308    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:11:38.403763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.403727    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-kube-api-access-49ztv\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-kube-api-access-49ztv\") pod \"9810b119-3076-49fd-9c52-140de7ec9b1f\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") "
Nov 03 00:11:38.403925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.403784    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9810b119-3076-49fd-9c52-140de7ec9b1f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9810b119-3076-49fd-9c52-140de7ec9b1f-local-certs\") pod \"9810b119-3076-49fd-9c52-140de7ec9b1f\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") "
Nov 03 00:11:38.403925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.403824    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-token\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-token\") pod \"9810b119-3076-49fd-9c52-140de7ec9b1f\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") "
Nov 03 00:11:38.403925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.403841    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-ca-configmap\") pod \"9810b119-3076-49fd-9c52-140de7ec9b1f\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") "
Nov 03 00:11:38.403925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.403898    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-dns-cert\") pod \"9810b119-3076-49fd-9c52-140de7ec9b1f\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") "
Nov 03 00:11:38.403925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.403925    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-kubeconfig\") pod \"9810b119-3076-49fd-9c52-140de7ec9b1f\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") "
Nov 03 00:11:38.404150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.403943    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-cacerts\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-cacerts\") pod \"9810b119-3076-49fd-9c52-140de7ec9b1f\" (UID: \"9810b119-3076-49fd-9c52-140de7ec9b1f\") "
Nov 03 00:11:38.404549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.404517    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-ca-configmap" pod "9810b119-3076-49fd-9c52-140de7ec9b1f" (UID: "9810b119-3076-49fd-9c52-140de7ec9b1f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:11:38.407005 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.406972    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-dns-cert" pod "9810b119-3076-49fd-9c52-140de7ec9b1f" (UID: "9810b119-3076-49fd-9c52-140de7ec9b1f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:38.407131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.407078    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-kubeconfig" pod "9810b119-3076-49fd-9c52-140de7ec9b1f" (UID: "9810b119-3076-49fd-9c52-140de7ec9b1f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:38.407131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.407114    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-kube-api-access-49ztv" pod "9810b119-3076-49fd-9c52-140de7ec9b1f" (UID: "9810b119-3076-49fd-9c52-140de7ec9b1f"). InnerVolumeSpecName "kube-api-access-49ztv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:11:38.407249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.407115    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-token" pod "9810b119-3076-49fd-9c52-140de7ec9b1f" (UID: "9810b119-3076-49fd-9c52-140de7ec9b1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:11:38.407368 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.407277    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-cacerts" pod "9810b119-3076-49fd-9c52-140de7ec9b1f" (UID: "9810b119-3076-49fd-9c52-140de7ec9b1f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:38.407487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.407471    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9810b119-3076-49fd-9c52-140de7ec9b1f-local-certs" pod "9810b119-3076-49fd-9c52-140de7ec9b1f" (UID: "9810b119-3076-49fd-9c52-140de7ec9b1f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:11:38.504932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.504860    6000 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9810b119-3076-49fd-9c52-140de7ec9b1f-local-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:38.504932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.504884    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:38.504932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.504893    6000 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-ca-configmap\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:38.504932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.504902    6000 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-csr-dns-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:38.504932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.504911    6000 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-istio-kubeconfig\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:38.504932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.504920    6000 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9810b119-3076-49fd-9c52-140de7ec9b1f-cacerts\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:38.504932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.504927    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49ztv\" (UniqueName: \"kubernetes.io/projected/9810b119-3076-49fd-9c52-140de7ec9b1f-kube-api-access-49ztv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:38.916022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.915984    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:38.916201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.916065    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:11:38.979553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.979517    6000 generic.go:356] "Generic (PLEG): container finished" podID="9810b119-3076-49fd-9c52-140de7ec9b1f" containerID="01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0" exitCode=0
Nov 03 00:11:38.980030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.979595    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"
Nov 03 00:11:38.980030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.979602    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" event={"ID":"9810b119-3076-49fd-9c52-140de7ec9b1f","Type":"ContainerDied","Data":"01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0"}
Nov 03 00:11:38.980030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.979662    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" event={"ID":"9810b119-3076-49fd-9c52-140de7ec9b1f","Type":"ContainerDied","Data":"4d7c663811722306a2d7fd6ff7758fc3b9b7e29d03fe77b59d312584e7e3db51"}
Nov 03 00:11:38.980030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.979697    6000 scope.go:122] "RemoveContainer" containerID="01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0"
Nov 03 00:11:38.980803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.980670    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d9d5b6dc8-2n7g9"]
Nov 03 00:11:38.980980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.980851    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9" podUID="13edb37b-2005-4a92-9833-8850cdcb2157" containerName="manager" containerID="cri-o://9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa" gracePeriod=30
Nov 03 00:11:38.999683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:38.999637    6000 scope.go:122] "RemoveContainer" containerID="01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0"
Nov 03 00:11:39.001525 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:11:39.001495    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0\": container with ID starting with 01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0 not found: ID does not exist" containerID="01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0"
Nov 03 00:11:39.001650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.001533    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0"} err="failed to get container status \"01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0\": rpc error: code = NotFound desc = could not find container \"01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0\": container with ID starting with 01ace21a4ae21f5f92ea5a458e1b973723ed5a93494a0f1256b540b1b12853d0 not found: ID does not exist"
Nov 03 00:11:39.029026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.028998    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"]
Nov 03 00:11:39.033222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.033200    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv"]
Nov 03 00:11:39.240941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.240910    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:39.414780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.414741    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-webhook-cert\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-webhook-cert\") pod \"13edb37b-2005-4a92-9833-8850cdcb2157\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") "
Nov 03 00:11:39.414983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.414851    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13edb37b-2005-4a92-9833-8850cdcb2157-kube-api-access-ckgk2\" (UniqueName: \"kubernetes.io/projected/13edb37b-2005-4a92-9833-8850cdcb2157-kube-api-access-ckgk2\") pod \"13edb37b-2005-4a92-9833-8850cdcb2157\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") "
Nov 03 00:11:39.414983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.414913    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-metrics-certs\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-metrics-certs\") pod \"13edb37b-2005-4a92-9833-8850cdcb2157\" (UID: \"13edb37b-2005-4a92-9833-8850cdcb2157\") "
Nov 03 00:11:39.417890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.417855    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13edb37b-2005-4a92-9833-8850cdcb2157-kube-api-access-ckgk2" pod "13edb37b-2005-4a92-9833-8850cdcb2157" (UID: "13edb37b-2005-4a92-9833-8850cdcb2157"). InnerVolumeSpecName "kube-api-access-ckgk2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:11:39.418019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.417862    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-webhook-cert" pod "13edb37b-2005-4a92-9833-8850cdcb2157" (UID: "13edb37b-2005-4a92-9833-8850cdcb2157"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:39.418019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.417972    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-metrics-certs" pod "13edb37b-2005-4a92-9833-8850cdcb2157" (UID: "13edb37b-2005-4a92-9833-8850cdcb2157"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:11:39.516453 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.516379    6000 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-webhook-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:39.516453 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.516411    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckgk2\" (UniqueName: \"kubernetes.io/projected/13edb37b-2005-4a92-9833-8850cdcb2157-kube-api-access-ckgk2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:39.516453 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.516426    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13edb37b-2005-4a92-9833-8850cdcb2157-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:11:39.986521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.986485    6000 generic.go:356] "Generic (PLEG): container finished" podID="13edb37b-2005-4a92-9833-8850cdcb2157" containerID="9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa" exitCode=0
Nov 03 00:11:39.987046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.986541    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9"
Nov 03 00:11:39.987046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.986574    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9" event={"ID":"13edb37b-2005-4a92-9833-8850cdcb2157","Type":"ContainerDied","Data":"9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa"}
Nov 03 00:11:39.987046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.986650    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-d9d5b6dc8-2n7g9" event={"ID":"13edb37b-2005-4a92-9833-8850cdcb2157","Type":"ContainerDied","Data":"454e3847cb44b13e66bcfac8c2d32345fdc5f345f021a17084ef2064ab7bc30d"}
Nov 03 00:11:39.987046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:39.986685    6000 scope.go:122] "RemoveContainer" containerID="9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa"
Nov 03 00:11:40.004061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:40.004039    6000 scope.go:122] "RemoveContainer" containerID="9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa"
Nov 03 00:11:40.004459 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:11:40.004420    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa\": container with ID starting with 9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa not found: ID does not exist" containerID="9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa"
Nov 03 00:11:40.004537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:40.004466    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa"} err="failed to get container status \"9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa\": rpc error: code = NotFound desc = could not find container \"9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa\": container with ID starting with 9729e4b1e0f1a855ee04948e4134bec996a11d60bf5b16cd5bac1428eae097aa not found: ID does not exist"
Nov 03 00:11:40.033861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:40.033831    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-d9d5b6dc8-2n7g9"]
Nov 03 00:11:40.036996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:40.036961    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-d9d5b6dc8-2n7g9"]
Nov 03 00:11:40.566531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:40.566489    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13edb37b-2005-4a92-9833-8850cdcb2157" path="/var/lib/kubelet/pods/13edb37b-2005-4a92-9833-8850cdcb2157/volumes"
Nov 03 00:11:40.567088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:40.567071    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9810b119-3076-49fd-9c52-140de7ec9b1f" path="/var/lib/kubelet/pods/9810b119-3076-49fd-9c52-140de7ec9b1f/volumes"
Nov 03 00:11:43.274495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:43.274461    6000 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8n6rv container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.21:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:11:43.274872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:43.274525    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8n6rv" podUID="9810b119-3076-49fd-9c52-140de7ec9b1f" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.21:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:11:44.017543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:44.017510    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874" event={"ID":"6f731881-a91d-48e5-b881-8bee6561c7fb","Type":"ContainerStarted","Data":"b09b3b7ecd0252e400e1c40447412ce214cc273e313a191540163e0de4b8898d"}
Nov 03 00:11:44.017724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:44.017640    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:11:44.054576 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:44.054515    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874" podStartSLOduration=1.724383938 podStartE2EDuration="16.054497359s" totalImagesPullingTime="14.330113421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:11:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:11:44.053186456 +0000 UTC m=+341.608538474" watchObservedRunningTime="2026-11-03 00:11:44.054497359 +0000 UTC m=+341.609849383"
Nov 03 00:11:49.893468 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:11:49.893436    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:11:55.027398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:11:55.027363    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-hm874"
Nov 03 00:12:00.474342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.474301    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"]
Nov 03 00:12:00.474944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.474871    6000 cpu_manager.go:414] "containerMap: removing container" podUID="13edb37b-2005-4a92-9833-8850cdcb2157" containerName="manager"
Nov 03 00:12:00.474944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.474902    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13edb37b-2005-4a92-9833-8850cdcb2157" containerName="manager"
Nov 03 00:12:00.475068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.474962    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9810b119-3076-49fd-9c52-140de7ec9b1f" containerName="discovery"
Nov 03 00:12:00.475068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.474970    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9810b119-3076-49fd-9c52-140de7ec9b1f" containerName="discovery"
Nov 03 00:12:00.475068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.475057    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="13edb37b-2005-4a92-9833-8850cdcb2157" containerName="manager"
Nov 03 00:12:00.475214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.475077    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9810b119-3076-49fd-9c52-140de7ec9b1f" containerName="discovery"
Nov 03 00:12:00.485973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.485946    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.497527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.497499    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"]
Nov 03 00:12:00.500126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.500107    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-x7pjw\""
Nov 03 00:12:00.500403 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.500145    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 03 00:12:00.500671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.500652    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 03 00:12:00.500801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.500674    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 03 00:12:00.500905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.500701    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 03 00:12:00.593196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.593164    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-metrics-certs\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.593402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.593340    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-maas-api-tls\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.593402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.593367    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74249ca-09f8-4c26-acba-26518e4772e3-openshift-service-ca\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.593495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.593407    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwhgl\" (UniqueName: \"kubernetes.io/projected/d74249ca-09f8-4c26-acba-26518e4772e3-kube-api-access-cwhgl\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.694848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.694810    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-metrics-certs\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.695026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.694891    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-maas-api-tls\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.695026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.694909    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74249ca-09f8-4c26-acba-26518e4772e3-openshift-service-ca\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.695026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.694935    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwhgl\" (UniqueName: \"kubernetes.io/projected/d74249ca-09f8-4c26-acba-26518e4772e3-kube-api-access-cwhgl\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.695669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.695644    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74249ca-09f8-4c26-acba-26518e4772e3-openshift-service-ca\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.697439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.697414    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-metrics-certs\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.697495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.697470    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-maas-api-tls\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.712783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.712760    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwhgl\" (UniqueName: \"kubernetes.io/projected/d74249ca-09f8-4c26-acba-26518e4772e3-kube-api-access-cwhgl\") pod \"maas-api-869f44cc68-4z5lg\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") " pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:00.800773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:00.800709    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:01.111615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:01.111592    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"]
Nov 03 00:12:01.113645 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:12:01.113614    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd74249ca_09f8_4c26_acba_26518e4772e3.slice/crio-8c6a457f38df65eb7d27611c7e1f2541fbd7e7082de2ebcf0807d991dd5c5fa8 WatchSource:0}: Error finding container 8c6a457f38df65eb7d27611c7e1f2541fbd7e7082de2ebcf0807d991dd5c5fa8: Status 404 returned error can't find the container with id 8c6a457f38df65eb7d27611c7e1f2541fbd7e7082de2ebcf0807d991dd5c5fa8
Nov 03 00:12:02.060626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:02.060592    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:12:02.093450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:02.093408    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg" event={"ID":"d74249ca-09f8-4c26-acba-26518e4772e3","Type":"ContainerStarted","Data":"8c6a457f38df65eb7d27611c7e1f2541fbd7e7082de2ebcf0807d991dd5c5fa8"}
Nov 03 00:12:05.109243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:05.109204    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg" event={"ID":"d74249ca-09f8-4c26-acba-26518e4772e3","Type":"ContainerStarted","Data":"7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce"}
Nov 03 00:12:05.109658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:05.109316    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:05.146574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:05.146528    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg" podStartSLOduration=2.115957979 podStartE2EDuration="5.146513261s" totalImagesPullingTime="3.030555282s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:05.14557923 +0000 UTC m=+362.700931255" watchObservedRunningTime="2026-11-03 00:12:05.146513261 +0000 UTC m=+362.701865288"
Nov 03 00:12:11.119007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:11.118978    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:12.066757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.066676    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-7gw56"]
Nov 03 00:12:12.080728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.080702    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.089707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.089682    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-7gw56"]
Nov 03 00:12:12.199768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.199732    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqb58\" (UniqueName: \"kubernetes.io/projected/d3701479-3c25-4485-8630-2e25c3950943-kube-api-access-nqb58\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.200200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.199852    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-maas-api-tls\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.200200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.199878    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-metrics-certs\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.200200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.199899    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3701479-3c25-4485-8630-2e25c3950943-openshift-service-ca\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.301085 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.301053    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-maas-api-tls\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.301292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.301097    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-metrics-certs\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.301292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.301126    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3701479-3c25-4485-8630-2e25c3950943-openshift-service-ca\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.301292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.301200    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqb58\" (UniqueName: \"kubernetes.io/projected/d3701479-3c25-4485-8630-2e25c3950943-kube-api-access-nqb58\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.301898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.301867    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3701479-3c25-4485-8630-2e25c3950943-openshift-service-ca\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.303840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.303815    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-metrics-certs\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.303938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.303840    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-maas-api-tls\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.319289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.319215    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqb58\" (UniqueName: \"kubernetes.io/projected/d3701479-3c25-4485-8630-2e25c3950943-kube-api-access-nqb58\") pod \"maas-api-6675486f96-7gw56\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") " pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.393863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.393821    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:12.690214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:12.690188    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-7gw56"]
Nov 03 00:12:12.691399 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:12:12.691375    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3701479_3c25_4485_8630_2e25c3950943.slice/crio-1ef2b285a554d26845ecb1356ab1235a5d02ce946d5a770e9de30d703cb6e961 WatchSource:0}: Error finding container 1ef2b285a554d26845ecb1356ab1235a5d02ce946d5a770e9de30d703cb6e961: Status 404 returned error can't find the container with id 1ef2b285a554d26845ecb1356ab1235a5d02ce946d5a770e9de30d703cb6e961
Nov 03 00:12:13.144776 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:13.144739    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56" event={"ID":"d3701479-3c25-4485-8630-2e25c3950943","Type":"ContainerStarted","Data":"e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b"}
Nov 03 00:12:13.144975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:13.144796    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56" event={"ID":"d3701479-3c25-4485-8630-2e25c3950943","Type":"ContainerStarted","Data":"1ef2b285a554d26845ecb1356ab1235a5d02ce946d5a770e9de30d703cb6e961"}
Nov 03 00:12:13.144975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:13.144911    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:13.182490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:13.182438    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56" podStartSLOduration=0.876488216 podStartE2EDuration="1.182420085s" totalImagesPullingTime="305.931869ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:12:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:12:13.181035103 +0000 UTC m=+370.736387128" watchObservedRunningTime="2026-11-03 00:12:13.182420085 +0000 UTC m=+370.737772108"
Nov 03 00:12:19.154482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.154452    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 00:12:19.222651 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.222613    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"]
Nov 03 00:12:19.222865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.222842    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg" podUID="d74249ca-09f8-4c26-acba-26518e4772e3" containerName="maas-api" containerID="cri-o://7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce" gracePeriod=30
Nov 03 00:12:19.475594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.475572    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:19.571322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.571283    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74249ca-09f8-4c26-acba-26518e4772e3-kube-api-access-cwhgl\" (UniqueName: \"kubernetes.io/projected/d74249ca-09f8-4c26-acba-26518e4772e3-kube-api-access-cwhgl\") pod \"d74249ca-09f8-4c26-acba-26518e4772e3\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") "
Nov 03 00:12:19.571502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.571366    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d74249ca-09f8-4c26-acba-26518e4772e3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74249ca-09f8-4c26-acba-26518e4772e3-openshift-service-ca\") pod \"d74249ca-09f8-4c26-acba-26518e4772e3\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") "
Nov 03 00:12:19.571502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.571416    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-maas-api-tls\") pod \"d74249ca-09f8-4c26-acba-26518e4772e3\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") "
Nov 03 00:12:19.571502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.571435    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-metrics-certs\") pod \"d74249ca-09f8-4c26-acba-26518e4772e3\" (UID: \"d74249ca-09f8-4c26-acba-26518e4772e3\") "
Nov 03 00:12:19.571771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.571739    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d74249ca-09f8-4c26-acba-26518e4772e3-openshift-service-ca" pod "d74249ca-09f8-4c26-acba-26518e4772e3" (UID: "d74249ca-09f8-4c26-acba-26518e4772e3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:12:19.573797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.573772    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-metrics-certs" pod "d74249ca-09f8-4c26-acba-26518e4772e3" (UID: "d74249ca-09f8-4c26-acba-26518e4772e3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:12:19.573890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.573836    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-maas-api-tls" pod "d74249ca-09f8-4c26-acba-26518e4772e3" (UID: "d74249ca-09f8-4c26-acba-26518e4772e3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:12:19.573980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.573889    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74249ca-09f8-4c26-acba-26518e4772e3-kube-api-access-cwhgl" pod "d74249ca-09f8-4c26-acba-26518e4772e3" (UID: "d74249ca-09f8-4c26-acba-26518e4772e3"). InnerVolumeSpecName "kube-api-access-cwhgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:12:19.672774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.672745    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwhgl\" (UniqueName: \"kubernetes.io/projected/d74249ca-09f8-4c26-acba-26518e4772e3-kube-api-access-cwhgl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:12:19.672774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.672773    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d74249ca-09f8-4c26-acba-26518e4772e3-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:12:19.672774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.672783    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:12:19.672996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:19.672792    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d74249ca-09f8-4c26-acba-26518e4772e3-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:12:19.895099 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:12:19.895063    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:12:20.176936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.176854    6000 generic.go:356] "Generic (PLEG): container finished" podID="d74249ca-09f8-4c26-acba-26518e4772e3" containerID="7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce" exitCode=0
Nov 03 00:12:20.176936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.176900    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"
Nov 03 00:12:20.177383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.176941    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg" event={"ID":"d74249ca-09f8-4c26-acba-26518e4772e3","Type":"ContainerDied","Data":"7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce"}
Nov 03 00:12:20.177383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.176997    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg" event={"ID":"d74249ca-09f8-4c26-acba-26518e4772e3","Type":"ContainerDied","Data":"8c6a457f38df65eb7d27611c7e1f2541fbd7e7082de2ebcf0807d991dd5c5fa8"}
Nov 03 00:12:20.177383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.177022    6000 scope.go:122] "RemoveContainer" containerID="7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce"
Nov 03 00:12:20.186099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.186082    6000 scope.go:122] "RemoveContainer" containerID="7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce"
Nov 03 00:12:20.186399 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:12:20.186380    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce\": container with ID starting with 7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce not found: ID does not exist" containerID="7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce"
Nov 03 00:12:20.186440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.186412    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce"} err="failed to get container status \"7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce\": rpc error: code = NotFound desc = could not find container \"7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce\": container with ID starting with 7b0aee797207c700b2f2613bd6b5ec7b21354de352c010d59a0d1f0f9747f6ce not found: ID does not exist"
Nov 03 00:12:20.239660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.239622    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"]
Nov 03 00:12:20.248570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.248542    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-869f44cc68-4z5lg"]
Nov 03 00:12:20.565686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:12:20.565608    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74249ca-09f8-4c26-acba-26518e4772e3" path="/var/lib/kubelet/pods/d74249ca-09f8-4c26-acba-26518e4772e3/volumes"
Nov 03 00:12:49.895550 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:12:49.895518    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:13:08.836219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.836185    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"]
Nov 03 00:13:08.836651 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.836594    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d74249ca-09f8-4c26-acba-26518e4772e3" containerName="maas-api"
Nov 03 00:13:08.836651 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.836605    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74249ca-09f8-4c26-acba-26518e4772e3" containerName="maas-api"
Nov 03 00:13:08.836725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.836667    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74249ca-09f8-4c26-acba-26518e4772e3" containerName="maas-api"
Nov 03 00:13:08.839382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.839358    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:08.857376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.857349    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"]
Nov 03 00:13:08.867733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.867704    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 03 00:13:08.867931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.867820    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-rc846\""
Nov 03 00:13:08.867931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:08.867832    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 03 00:13:09.002004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.001970    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437a6cf4-c267-4907-95cf-be0e1c43d9cd-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-cgqpk\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:09.002160 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.002022    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4gr8\" (UniqueName: \"kubernetes.io/projected/437a6cf4-c267-4907-95cf-be0e1c43d9cd-kube-api-access-l4gr8\") pod \"ai-gateway-controller-5d8dddb676-cgqpk\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:09.103476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.103451    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4gr8\" (UniqueName: \"kubernetes.io/projected/437a6cf4-c267-4907-95cf-be0e1c43d9cd-kube-api-access-l4gr8\") pod \"ai-gateway-controller-5d8dddb676-cgqpk\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:09.103628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.103533    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437a6cf4-c267-4907-95cf-be0e1c43d9cd-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-cgqpk\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:09.106031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.106003    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437a6cf4-c267-4907-95cf-be0e1c43d9cd-metrics-certs\") pod \"ai-gateway-controller-5d8dddb676-cgqpk\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:09.122533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.122504    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4gr8\" (UniqueName: \"kubernetes.io/projected/437a6cf4-c267-4907-95cf-be0e1c43d9cd-kube-api-access-l4gr8\") pod \"ai-gateway-controller-5d8dddb676-cgqpk\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") " pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:09.151183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.151154    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:09.446383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:09.446294    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"]
Nov 03 00:13:09.450710 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:13:09.450670    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437a6cf4_c267_4907_95cf_be0e1c43d9cd.slice/crio-3c674d1b3e68aa94411859c7e07b8765a3ed4eb24abc0ee598454912fbcf1ca7 WatchSource:0}: Error finding container 3c674d1b3e68aa94411859c7e07b8765a3ed4eb24abc0ee598454912fbcf1ca7: Status 404 returned error can't find the container with id 3c674d1b3e68aa94411859c7e07b8765a3ed4eb24abc0ee598454912fbcf1ca7
Nov 03 00:13:10.384672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:10.384637    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk" event={"ID":"437a6cf4-c267-4907-95cf-be0e1c43d9cd","Type":"ContainerStarted","Data":"3c674d1b3e68aa94411859c7e07b8765a3ed4eb24abc0ee598454912fbcf1ca7"}
Nov 03 00:13:11.390462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:11.390425    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk" event={"ID":"437a6cf4-c267-4907-95cf-be0e1c43d9cd","Type":"ContainerStarted","Data":"2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4"}
Nov 03 00:13:11.390867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:11.390502    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:11.428121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:11.428069    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk" podStartSLOduration=1.838233993 podStartE2EDuration="3.428054878s" totalImagesPullingTime="1.589820885s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:11.426717025 +0000 UTC m=+428.982069050" watchObservedRunningTime="2026-11-03 00:13:11.428054878 +0000 UTC m=+428.983406902"
Nov 03 00:13:13.188521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.188487    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"]
Nov 03 00:13:13.191050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.191025    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.209901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.209874    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Nov 03 00:13:13.223163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.223128    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"]
Nov 03 00:13:13.241024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.240987    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t65qw\" (UniqueName: \"kubernetes.io/projected/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-kube-api-access-t65qw\") pod \"payload-pre-processing-7b98bb678c-4m2w6\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.241179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.241101    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-4m2w6\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.342462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.342418    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t65qw\" (UniqueName: \"kubernetes.io/projected/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-kube-api-access-t65qw\") pod \"payload-pre-processing-7b98bb678c-4m2w6\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.342654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.342524    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-4m2w6\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.343301 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.343283    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-4m2w6\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.361157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.361125    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t65qw\" (UniqueName: \"kubernetes.io/projected/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-kube-api-access-t65qw\") pod \"payload-pre-processing-7b98bb678c-4m2w6\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.502560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.502473    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:13.796291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:13.796230    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"]
Nov 03 00:13:13.798153 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:13:13.798126    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d46ad2b_b643_4bb4_a01d_51c9c1f3c2fe.slice/crio-1d54e908f522dff7e77d2e957adbadbf50af605982a422a3ae81dd0f4535526b WatchSource:0}: Error finding container 1d54e908f522dff7e77d2e957adbadbf50af605982a422a3ae81dd0f4535526b: Status 404 returned error can't find the container with id 1d54e908f522dff7e77d2e957adbadbf50af605982a422a3ae81dd0f4535526b
Nov 03 00:13:14.406760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:14.406724    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6" event={"ID":"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe","Type":"ContainerStarted","Data":"1d54e908f522dff7e77d2e957adbadbf50af605982a422a3ae81dd0f4535526b"}
Nov 03 00:13:16.417928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:16.417892    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6" event={"ID":"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe","Type":"ContainerStarted","Data":"27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2"}
Nov 03 00:13:16.418372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:16.418009    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:16.456214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:16.456168    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6" podStartSLOduration=1.5825895779999999 podStartE2EDuration="3.456153431s" totalImagesPullingTime="1.873563853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:16.455116478 +0000 UTC m=+434.010468502" watchObservedRunningTime="2026-11-03 00:13:16.456153431 +0000 UTC m=+434.011505498"
Nov 03 00:13:19.895682 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:13:19.895641    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:13:22.398393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.398363    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:22.698325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.698221    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"]
Nov 03 00:13:22.700760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.700743    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:22.721168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.721145    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"]
Nov 03 00:13:22.727320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.727297    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5lpv\" (UniqueName: \"kubernetes.io/projected/ec8b5850-10d2-4e94-96fc-834119b90561-kube-api-access-d5lpv\") pod \"ai-gateway-controller-77bbb667f4-ps8vw\" (UID: \"ec8b5850-10d2-4e94-96fc-834119b90561\") " pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:22.727432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.727338    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8b5850-10d2-4e94-96fc-834119b90561-metrics-certs\") pod \"ai-gateway-controller-77bbb667f4-ps8vw\" (UID: \"ec8b5850-10d2-4e94-96fc-834119b90561\") " pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:22.828327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.828297    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5lpv\" (UniqueName: \"kubernetes.io/projected/ec8b5850-10d2-4e94-96fc-834119b90561-kube-api-access-d5lpv\") pod \"ai-gateway-controller-77bbb667f4-ps8vw\" (UID: \"ec8b5850-10d2-4e94-96fc-834119b90561\") " pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:22.828521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.828337    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8b5850-10d2-4e94-96fc-834119b90561-metrics-certs\") pod \"ai-gateway-controller-77bbb667f4-ps8vw\" (UID: \"ec8b5850-10d2-4e94-96fc-834119b90561\") " pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:22.830823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.830803    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec8b5850-10d2-4e94-96fc-834119b90561-metrics-certs\") pod \"ai-gateway-controller-77bbb667f4-ps8vw\" (UID: \"ec8b5850-10d2-4e94-96fc-834119b90561\") " pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:22.846677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:22.846654    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5lpv\" (UniqueName: \"kubernetes.io/projected/ec8b5850-10d2-4e94-96fc-834119b90561-kube-api-access-d5lpv\") pod \"ai-gateway-controller-77bbb667f4-ps8vw\" (UID: \"ec8b5850-10d2-4e94-96fc-834119b90561\") " pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:23.012109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:23.012020    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:23.307322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:23.307296    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"]
Nov 03 00:13:23.308825 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:13:23.308797    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec8b5850_10d2_4e94_96fc_834119b90561.slice/crio-1b2e507d4c4abee6a6371901b750306f55d8764a8ed1c92ef71e6019b47aef88 WatchSource:0}: Error finding container 1b2e507d4c4abee6a6371901b750306f55d8764a8ed1c92ef71e6019b47aef88: Status 404 returned error can't find the container with id 1b2e507d4c4abee6a6371901b750306f55d8764a8ed1c92ef71e6019b47aef88
Nov 03 00:13:23.447973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:23.447935    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw" event={"ID":"ec8b5850-10d2-4e94-96fc-834119b90561","Type":"ContainerStarted","Data":"1b2e507d4c4abee6a6371901b750306f55d8764a8ed1c92ef71e6019b47aef88"}
Nov 03 00:13:24.454177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:24.454141    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw" event={"ID":"ec8b5850-10d2-4e94-96fc-834119b90561","Type":"ContainerStarted","Data":"f0ce3552d13fd4fabc17b49d821dcc20fe6d691b6daa20f48a82bee8d648b60a"}
Nov 03 00:13:24.454593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:24.454296    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:24.491704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:24.491599    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw" podStartSLOduration=2.187351975 podStartE2EDuration="2.491583989s" totalImagesPullingTime="304.232014ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:24.490561553 +0000 UTC m=+442.045913577" watchObservedRunningTime="2026-11-03 00:13:24.491583989 +0000 UTC m=+442.046936014"
Nov 03 00:13:27.425152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:27.425118    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 00:13:35.461115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.461084    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-77bbb667f4-ps8vw"
Nov 03 00:13:35.521541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.521505    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"]
Nov 03 00:13:35.521823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.521780    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk" podUID="437a6cf4-c267-4907-95cf-be0e1c43d9cd" containerName="manager" containerID="cri-o://2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4" gracePeriod=30
Nov 03 00:13:35.764392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.764369    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:35.847448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.847416    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/437a6cf4-c267-4907-95cf-be0e1c43d9cd-kube-api-access-l4gr8\" (UniqueName: \"kubernetes.io/projected/437a6cf4-c267-4907-95cf-be0e1c43d9cd-kube-api-access-l4gr8\") pod \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") "
Nov 03 00:13:35.847615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.847470    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/437a6cf4-c267-4907-95cf-be0e1c43d9cd-metrics-certs\" (UniqueName: \"kubernetes.io/secret/437a6cf4-c267-4907-95cf-be0e1c43d9cd-metrics-certs\") pod \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\" (UID: \"437a6cf4-c267-4907-95cf-be0e1c43d9cd\") "
Nov 03 00:13:35.849677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.849647    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/437a6cf4-c267-4907-95cf-be0e1c43d9cd-metrics-certs" pod "437a6cf4-c267-4907-95cf-be0e1c43d9cd" (UID: "437a6cf4-c267-4907-95cf-be0e1c43d9cd"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:13:35.849792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.849698    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/437a6cf4-c267-4907-95cf-be0e1c43d9cd-kube-api-access-l4gr8" pod "437a6cf4-c267-4907-95cf-be0e1c43d9cd" (UID: "437a6cf4-c267-4907-95cf-be0e1c43d9cd"). InnerVolumeSpecName "kube-api-access-l4gr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:13:35.949087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.949055    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4gr8\" (UniqueName: \"kubernetes.io/projected/437a6cf4-c267-4907-95cf-be0e1c43d9cd-kube-api-access-l4gr8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:13:35.949087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:35.949082    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/437a6cf4-c267-4907-95cf-be0e1c43d9cd-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:13:36.504419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.504381    6000 generic.go:356] "Generic (PLEG): container finished" podID="437a6cf4-c267-4907-95cf-be0e1c43d9cd" containerID="2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4" exitCode=0
Nov 03 00:13:36.504860 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.504431    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk" event={"ID":"437a6cf4-c267-4907-95cf-be0e1c43d9cd","Type":"ContainerDied","Data":"2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4"}
Nov 03 00:13:36.504860 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.504444    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"
Nov 03 00:13:36.504860 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.504508    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-5d8dddb676-cgqpk" event={"ID":"437a6cf4-c267-4907-95cf-be0e1c43d9cd","Type":"ContainerDied","Data":"3c674d1b3e68aa94411859c7e07b8765a3ed4eb24abc0ee598454912fbcf1ca7"}
Nov 03 00:13:36.504860 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.504532    6000 scope.go:122] "RemoveContainer" containerID="2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4"
Nov 03 00:13:36.514515 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.514495    6000 scope.go:122] "RemoveContainer" containerID="2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4"
Nov 03 00:13:36.514796 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:13:36.514776    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4\": container with ID starting with 2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4 not found: ID does not exist" containerID="2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4"
Nov 03 00:13:36.514858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.514812    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4"} err="failed to get container status \"2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4\": rpc error: code = NotFound desc = could not find container \"2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4\": container with ID starting with 2fefd99286cf5e38b89336595e499d956399d4c3bf0a3c284c6658169b36a1f4 not found: ID does not exist"
Nov 03 00:13:36.552108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.552077    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"]
Nov 03 00:13:36.565809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:36.565778    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-5d8dddb676-cgqpk"]
Nov 03 00:13:38.564731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:38.564692    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="437a6cf4-c267-4907-95cf-be0e1c43d9cd" path="/var/lib/kubelet/pods/437a6cf4-c267-4907-95cf-be0e1c43d9cd/volumes"
Nov 03 00:13:43.195813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.195736    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"]
Nov 03 00:13:43.196356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.196337    6000 cpu_manager.go:414] "containerMap: removing container" podUID="437a6cf4-c267-4907-95cf-be0e1c43d9cd" containerName="manager"
Nov 03 00:13:43.196454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.196359    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="437a6cf4-c267-4907-95cf-be0e1c43d9cd" containerName="manager"
Nov 03 00:13:43.196454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.196450    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="437a6cf4-c267-4907-95cf-be0e1c43d9cd" containerName="manager"
Nov 03 00:13:43.198560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.198538    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.215946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.215923    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 03 00:13:43.216081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.216018    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 03 00:13:43.216144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.216091    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 03 00:13:43.216188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.216163    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-fm44c\""
Nov 03 00:13:43.217084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.217063    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"]
Nov 03 00:13:43.324530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.324493    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2gld\" (UniqueName: \"kubernetes.io/projected/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-kube-api-access-z2gld\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.324673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.324545    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.324673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.324652    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.324762 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.324684    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.324793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.324765    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.324827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.324814    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.425642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.425607    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.425811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.425665    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.425811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.425694    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.425811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.425739    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.425912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.425872    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.425955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.425929    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2gld\" (UniqueName: \"kubernetes.io/projected/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-kube-api-access-z2gld\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.426219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.426186    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.426219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.426214    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.426620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.426337    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.428209 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.428188    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.428394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.428377    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.444119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.444095    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2gld\" (UniqueName: \"kubernetes.io/projected/04cc1bb4-bbe7-468b-9e49-8df72c661ee1-kube-api-access-z2gld\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29\" (UID: \"04cc1bb4-bbe7-468b-9e49-8df72c661ee1\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.510729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.510643    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:43.806678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:43.806650    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"]
Nov 03 00:13:43.808757 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:13:43.808727    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cc1bb4_bbe7_468b_9e49_8df72c661ee1.slice/crio-25736636a825e39a70d3a6c435ecd2c6575a592b82dd18013360f5426cbe9194 WatchSource:0}: Error finding container 25736636a825e39a70d3a6c435ecd2c6575a592b82dd18013360f5426cbe9194: Status 404 returned error can't find the container with id 25736636a825e39a70d3a6c435ecd2c6575a592b82dd18013360f5426cbe9194
Nov 03 00:13:44.169524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:44.168890    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:13:44.543169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:44.543083    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29" event={"ID":"04cc1bb4-bbe7-468b-9e49-8df72c661ee1","Type":"ContainerStarted","Data":"25736636a825e39a70d3a6c435ecd2c6575a592b82dd18013360f5426cbe9194"}
Nov 03 00:13:45.548290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:45.548239    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29" event={"ID":"04cc1bb4-bbe7-468b-9e49-8df72c661ee1","Type":"ContainerStarted","Data":"ecadfc6096486383bd6e43e0c1c2990b5a91558bd86c2f793c45091dbded0a96"}
Nov 03 00:13:45.586753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:45.586703    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29" podStartSLOduration=1.5001698399999999 podStartE2EDuration="2.58668548s" totalImagesPullingTime="1.08651564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:45.585882737 +0000 UTC m=+463.141234774" watchObservedRunningTime="2026-11-03 00:13:45.58668548 +0000 UTC m=+463.142037504"
Nov 03 00:13:46.511024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:46.510994    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:47.839390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:47.839357    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:13:48.170041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.170012    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"]
Nov 03 00:13:48.173771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.173750    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.189374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.189349    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 03 00:13:48.202599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.202573    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"]
Nov 03 00:13:48.271215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.271183    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.271215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.271213    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.271445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.271240    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.271445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.271402    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.271530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.271484    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45l8c\" (UniqueName: \"kubernetes.io/projected/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-kube-api-access-45l8c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.271530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.271512    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.372556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.372518    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45l8c\" (UniqueName: \"kubernetes.io/projected/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-kube-api-access-45l8c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.372556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.372556    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.372804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.372586    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.372804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.372601    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.372804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.372625    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.372804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.372671    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.373062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.373037    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.373144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.373099    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.373144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.373114    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.375116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.375087    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.375287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.375239    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.390810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.390787    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45l8c\" (UniqueName: \"kubernetes.io/projected/f6adde23-6bf2-4a92-8002-a6e093c1c2ba-kube-api-access-45l8c\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-m2dbv\" (UID: \"f6adde23-6bf2-4a92-8002-a6e093c1c2ba\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.497721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.497648    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:48.791866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:48.791840    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"]
Nov 03 00:13:48.793757 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:13:48.793719    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6adde23_6bf2_4a92_8002_a6e093c1c2ba.slice/crio-ec390526dbb1ef056f6d838d7b08b5399de564dd1717805c3929b75e24e8180d WatchSource:0}: Error finding container ec390526dbb1ef056f6d838d7b08b5399de564dd1717805c3929b75e24e8180d: Status 404 returned error can't find the container with id ec390526dbb1ef056f6d838d7b08b5399de564dd1717805c3929b75e24e8180d
Nov 03 00:13:49.575166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:49.575073    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv" event={"ID":"f6adde23-6bf2-4a92-8002-a6e093c1c2ba","Type":"ContainerStarted","Data":"c63296f6b5b7766697b70aea354a14a609fa60212819fcc77812de2f668dc76e"}
Nov 03 00:13:49.575166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:49.575135    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv" event={"ID":"f6adde23-6bf2-4a92-8002-a6e093c1c2ba","Type":"ContainerStarted","Data":"ec390526dbb1ef056f6d838d7b08b5399de564dd1717805c3929b75e24e8180d"}
Nov 03 00:13:49.614742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:49.614691    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv" podStartSLOduration=1.269528373 podStartE2EDuration="1.614675369s" totalImagesPullingTime="345.146996ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:49.613731235 +0000 UTC m=+467.169083259" watchObservedRunningTime="2026-11-03 00:13:49.614675369 +0000 UTC m=+467.170027390"
Nov 03 00:13:49.894817 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:13:49.894783    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:13:50.497796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:50.497758    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:50.846602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:50.846532    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:13:52.525654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.525621    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lfqwv"]
Nov 03 00:13:52.529513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.529490    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.543268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.543226    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\""
Nov 03 00:13:52.555988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.555961    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lfqwv"]
Nov 03 00:13:52.714450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.714415    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.714611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.714473    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.714611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.714545    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njsjh\" (UniqueName: \"kubernetes.io/projected/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-kube-api-access-njsjh\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.714611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.714591    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.714752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.714620    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-home\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.714752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.714711    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815130    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815176    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-home\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815202    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815246    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815325    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815356    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njsjh\" (UniqueName: \"kubernetes.io/projected/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-kube-api-access-njsjh\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815673    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-home\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815757    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.815964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.815864    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.817921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.817895    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.818054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.818037    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.833611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.833587    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njsjh\" (UniqueName: \"kubernetes.io/projected/f61b18a7-80a7-431a-9a91-ac141ff2bcdf-kube-api-access-njsjh\") pod \"echo-simulator-kserve-64b6f8f96c-lfqwv\" (UID: \"f61b18a7-80a7-431a-9a91-ac141ff2bcdf\") " pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:52.844469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:52.844450    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:53.139080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:53.139051    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-lfqwv"]
Nov 03 00:13:53.140269 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:13:53.140222    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf61b18a7_80a7_431a_9a91_ac141ff2bcdf.slice/crio-163d1f23f3cb635d0eaee9fb29577e76d440cf0c33c571c53d7eea8604e9b306 WatchSource:0}: Error finding container 163d1f23f3cb635d0eaee9fb29577e76d440cf0c33c571c53d7eea8604e9b306: Status 404 returned error can't find the container with id 163d1f23f3cb635d0eaee9fb29577e76d440cf0c33c571c53d7eea8604e9b306
Nov 03 00:13:53.141684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:53.141667    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:13:53.512734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:53.511910    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:53.527802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:53.527771    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:53.593668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:53.593637    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv" event={"ID":"f61b18a7-80a7-431a-9a91-ac141ff2bcdf","Type":"ContainerStarted","Data":"163d1f23f3cb635d0eaee9fb29577e76d440cf0c33c571c53d7eea8604e9b306"}
Nov 03 00:13:53.605647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:53.605620    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29"
Nov 03 00:13:55.168779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.168743    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"]
Nov 03 00:13:55.172470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.172446    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.190448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.190424    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"]
Nov 03 00:13:55.197516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.197487    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 03 00:13:55.238469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.238444    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.238604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.238479    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.238604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.238510    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.238705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.238691    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdvbx\" (UniqueName: \"kubernetes.io/projected/772fdde9-c1c8-47a0-ada4-d34e8033de8b-kube-api-access-cdvbx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.238758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.238719    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.238758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.238750    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772fdde9-c1c8-47a0-ada4-d34e8033de8b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339145    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339183    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339285    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdvbx\" (UniqueName: \"kubernetes.io/projected/772fdde9-c1c8-47a0-ada4-d34e8033de8b-kube-api-access-cdvbx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339318    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339360    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772fdde9-c1c8-47a0-ada4-d34e8033de8b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339403    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339639    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339697    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.339781 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.339740    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.341656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.341630    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/772fdde9-c1c8-47a0-ada4-d34e8033de8b-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.342008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.341987    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/772fdde9-c1c8-47a0-ada4-d34e8033de8b-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.358235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.358213    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdvbx\" (UniqueName: \"kubernetes.io/projected/772fdde9-c1c8-47a0-ada4-d34e8033de8b-kube-api-access-cdvbx\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk\" (UID: \"772fdde9-c1c8-47a0-ada4-d34e8033de8b\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.488115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.488042    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:55.604360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.604324    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv" event={"ID":"f61b18a7-80a7-431a-9a91-ac141ff2bcdf","Type":"ContainerStarted","Data":"dad794a5b41d7baa61ebec550fda40dcb5259fc3cc82da66c2c65421135242c4"}
Nov 03 00:13:55.643849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.643799    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv" podStartSLOduration=2.093880498 podStartE2EDuration="3.643785218s" totalImagesPullingTime="1.54990472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:55.641980876 +0000 UTC m=+473.197332904" watchObservedRunningTime="2026-11-03 00:13:55.643785218 +0000 UTC m=+473.199137241"
Nov 03 00:13:55.785122 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.785087    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"]
Nov 03 00:13:55.786679 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:13:55.786651    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod772fdde9_c1c8_47a0_ada4_d34e8033de8b.slice/crio-912c9015f38d7d831a55cfd8c6280ce6c9cf2ad034fdc1d28e5b9962fc9ff15a WatchSource:0}: Error finding container 912c9015f38d7d831a55cfd8c6280ce6c9cf2ad034fdc1d28e5b9962fc9ff15a: Status 404 returned error can't find the container with id 912c9015f38d7d831a55cfd8c6280ce6c9cf2ad034fdc1d28e5b9962fc9ff15a
Nov 03 00:13:55.844686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:55.844657    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:13:56.610798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:56.610765    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk" event={"ID":"772fdde9-c1c8-47a0-ada4-d34e8033de8b","Type":"ContainerStarted","Data":"4b8a1fe3d6d1ba6888a81bef9ade7372330971443b4010f9a738044e76869f37"}
Nov 03 00:13:56.611215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:56.610826    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk" event={"ID":"772fdde9-c1c8-47a0-ada4-d34e8033de8b","Type":"ContainerStarted","Data":"912c9015f38d7d831a55cfd8c6280ce6c9cf2ad034fdc1d28e5b9962fc9ff15a"}
Nov 03 00:13:56.657919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:56.657873    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk" podStartSLOduration=1.437811991 podStartE2EDuration="1.657859127s" totalImagesPullingTime="220.047136ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:13:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:13:56.656441531 +0000 UTC m=+474.211793555" watchObservedRunningTime="2026-11-03 00:13:56.657859127 +0000 UTC m=+474.213211150"
Nov 03 00:13:56.714881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:56.714846    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:13:57.488159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:57.488133    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:13:57.619529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:57.619503    6000 generic.go:356] "Generic (PLEG): container finished" podID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerID="4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec" exitCode=1
Nov 03 00:13:57.619912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:57.619587    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" event={"ID":"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5","Type":"ContainerDied","Data":"4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec"}
Nov 03 00:13:57.620111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:57.620095    6000 scope.go:122] "RemoveContainer" containerID="4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec"
Nov 03 00:13:58.498364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:58.498331    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:58.511073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:58.511052    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:13:58.626495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:58.626459    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" event={"ID":"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5","Type":"ContainerStarted","Data":"d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab"}
Nov 03 00:13:58.626997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:58.626955    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:13:58.638085 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:13:58.638057    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv"
Nov 03 00:14:00.229820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:00.229787    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:14:02.845563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:02.845528    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:14:02.858047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:02.857968    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:14:03.660888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:03.660860    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-lfqwv"
Nov 03 00:14:05.489046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:05.489008    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:14:05.501416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:05.501389    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:14:05.670010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:05.669984    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk"
Nov 03 00:14:09.632393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:09.632365    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 00:14:12.199516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.199468    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"]
Nov 03 00:14:12.204591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.204569    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.218401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.218381    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v9b8n\""
Nov 03 00:14:12.231596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.231577    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"]
Nov 03 00:14:12.294872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.294830    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.294984    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.295020    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54cd764-315b-4333-960e-1b442ed0ab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.295043    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.295065    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54cd764-315b-4333-960e-1b442ed0ab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.295198    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.295234    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.295308    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djz2r\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-kube-api-access-djz2r\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.295373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.295331    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396068    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396130    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396157    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54cd764-315b-4333-960e-1b442ed0ab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396177    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396198    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54cd764-315b-4333-960e-1b442ed0ab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396244    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396303    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396342    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djz2r\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-kube-api-access-djz2r\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396367    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396576    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396639    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.396973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396947    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.397130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.396954    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.397130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.397041    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54cd764-315b-4333-960e-1b442ed0ab93-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.399181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.399161    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.399623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.399597    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54cd764-315b-4333-960e-1b442ed0ab93-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.414782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.414760    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djz2r\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-kube-api-access-djz2r\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.414862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.414794    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-dqqvx\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.517598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.517544    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:12.837751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:12.837727    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"]
Nov 03 00:14:12.839067 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:14:12.839039    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd54cd764_315b_4333_960e_1b442ed0ab93.slice/crio-40aa4169fceff3d3bf846484883a3ff9d94e20844da73e0d967298f8a45793c2 WatchSource:0}: Error finding container 40aa4169fceff3d3bf846484883a3ff9d94e20844da73e0d967298f8a45793c2: Status 404 returned error can't find the container with id 40aa4169fceff3d3bf846484883a3ff9d94e20844da73e0d967298f8a45793c2
Nov 03 00:14:13.697285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:13.697212    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" event={"ID":"d54cd764-315b-4333-960e-1b442ed0ab93","Type":"ContainerStarted","Data":"40aa4169fceff3d3bf846484883a3ff9d94e20844da73e0d967298f8a45793c2"}
Nov 03 00:14:15.443039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:15.443009    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:14:15.443318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:15.443076    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:14:15.443318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:15.443108    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:14:15.710449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:15.710355    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" event={"ID":"d54cd764-315b-4333-960e-1b442ed0ab93","Type":"ContainerStarted","Data":"9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3"}
Nov 03 00:14:15.771921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:15.771872    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podStartSLOduration=1.170553494 podStartE2EDuration="3.771859324s" totalImagesPullingTime="2.60130583s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:15.760073344 +0000 UTC m=+493.315425379" watchObservedRunningTime="2026-11-03 00:14:15.771859324 +0000 UTC m=+493.327211348"
Nov 03 00:14:16.518076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:16.518041    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:16.519526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:16.519498    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 00:14:16.519592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:16.519544    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 00:14:17.518970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:17.518931    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 00:14:17.519399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:17.518993    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 00:14:18.518639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:18.518600    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 00:14:18.518846 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:18.518669    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 00:14:19.518953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:19.518901    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 00:14:19.519353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:19.518978    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 00:14:19.915324 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:14:19.915282    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:14:20.517990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:20.517952    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 00:14:20.518210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:20.518008    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 00:14:21.518699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:21.518658    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 00:14:21.519095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:21.518723    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 00:14:21.608062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:21.608028    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:14:22.518339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:22.518307    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 03 00:14:22.518514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:22.518360    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 03 00:14:24.518197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:24.518103    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:14:24.518197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:24.518170    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:14:25.559402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:25.559363    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:14:25.559794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:25.559422    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:14:25.891351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:25.891317    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:25.891649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:25.891618    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:25.892346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:25.892328    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 00:14:27.256735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.256698    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"]
Nov 03 00:14:27.264678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.264652    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.292053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.292019    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"]
Nov 03 00:14:27.346411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346379    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86096906-704d-455f-aab8-4b852fb2bba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346418    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346508    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346529    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346551    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsq9m\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-kube-api-access-wsq9m\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346583    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346759    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346788    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86096906-704d-455f-aab8-4b852fb2bba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.346920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.346892    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448358    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86096906-704d-455f-aab8-4b852fb2bba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448399    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448449    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448471    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448496    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsq9m\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-kube-api-access-wsq9m\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448653    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448776    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448805    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86096906-704d-455f-aab8-4b852fb2bba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448869    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.448894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448880    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.449047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.448913    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.449047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.449009    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.449157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.449116    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.449444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.449426    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86096906-704d-455f-aab8-4b852fb2bba6-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.451169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.451148    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86096906-704d-455f-aab8-4b852fb2bba6-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.451296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.451246    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.466745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.466719    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsq9m\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-kube-api-access-wsq9m\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.467419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.467398    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-kjlpp\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.579625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.579536    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:27.888497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.888470    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"]
Nov 03 00:14:27.889734 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:14:27.889712    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86096906_704d_455f_aab8_4b852fb2bba6.slice/crio-b00980bacaf4b256e1fbd0a9172e6da4a70a72c2b970958b33bf1a64e64ef8bc WatchSource:0}: Error finding container b00980bacaf4b256e1fbd0a9172e6da4a70a72c2b970958b33bf1a64e64ef8bc: Status 404 returned error can't find the container with id b00980bacaf4b256e1fbd0a9172e6da4a70a72c2b970958b33bf1a64e64ef8bc
Nov 03 00:14:27.892871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.892844    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:14:27.892956 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.892907    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:14:27.892956 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:27.892935    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:14:28.777611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:28.777573    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" event={"ID":"86096906-704d-455f-aab8-4b852fb2bba6","Type":"ContainerStarted","Data":"9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6"}
Nov 03 00:14:28.778016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:28.777631    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" event={"ID":"86096906-704d-455f-aab8-4b852fb2bba6","Type":"ContainerStarted","Data":"b00980bacaf4b256e1fbd0a9172e6da4a70a72c2b970958b33bf1a64e64ef8bc"}
Nov 03 00:14:28.856993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:28.856943    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podStartSLOduration=1.856931136 podStartE2EDuration="1.856931136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:14:28.845576445 +0000 UTC m=+506.400928481" watchObservedRunningTime="2026-11-03 00:14:28.856931136 +0000 UTC m=+506.412283160"
Nov 03 00:14:29.580114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:29.580078    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:29.581587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:29.581554    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 00:14:29.581712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:29.581603    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 00:14:30.580496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:30.580463    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 00:14:30.580916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:30.580512    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 00:14:31.580320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:31.580283    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 00:14:31.580503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:31.580357    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 00:14:32.580581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:32.580548    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 00:14:32.580986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:32.580610    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 00:14:33.579969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:33.579933    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 00:14:33.580150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:33.579989    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 00:14:34.580777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:34.580734    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 03 00:14:34.581282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:34.580810    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 03 00:14:36.148156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:36.148121    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:36.148617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:36.148489    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:36.149492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:36.149472    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:14:49.897495 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:14:49.897447    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:14:59.401702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.401660    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"]
Nov 03 00:14:59.406052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.406031    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.424402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.424376    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a96a129b-openshift-default-dockercfg-tgvth\""
Nov 03 00:14:59.437509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.437482    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"]
Nov 03 00:14:59.476916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.476873    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-data\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.476940    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-credential-socket\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.477055    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvrb\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-kube-api-access-9dvrb\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.477081    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-envoy\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.477110    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-certs\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.477130    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-podinfo\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.477250    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-token\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.477354    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istiod-ca-cert\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.477459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.477400    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-socket\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.577988    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-data\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578041    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-credential-socket\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578074    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvrb\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-kube-api-access-9dvrb\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578096    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-envoy\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578121    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-certs\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578142    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-podinfo\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578175    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-token\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578215    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istiod-ca-cert\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578453    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-socket\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578472    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-data\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578546    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-certs\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578764    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-credential-socket\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578787    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-socket\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.578964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.578938    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istiod-ca-cert\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.581200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.581169    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-envoy\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.581333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.581317    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-podinfo\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.598512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.598488    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-token\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.598608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.598583    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvrb\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-kube-api-access-9dvrb\") pod \"e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") " pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:14:59.720695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:14:59.720607    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:00.061313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.061284    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"]
Nov 03 00:15:00.062692 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:00.062666    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ea1a000_9bcb_49d1_9e72_6ebe27cb3cd6.slice/crio-7f0609531175ebbaca6fecc173e252d25b706bbb89c45c8b011a77b86a8a642b WatchSource:0}: Error finding container 7f0609531175ebbaca6fecc173e252d25b706bbb89c45c8b011a77b86a8a642b: Status 404 returned error can't find the container with id 7f0609531175ebbaca6fecc173e252d25b706bbb89c45c8b011a77b86a8a642b
Nov 03 00:15:00.065573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.065525    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:00.065716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.065683    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:00.065807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.065786    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:00.865354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.865316    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"]
Nov 03 00:15:00.870244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.870219    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:00.890191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.890152    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-19590f89-openshift-default-dockercfg-sxslb\""
Nov 03 00:15:00.900886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.900860    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"]
Nov 03 00:15:00.952062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.952026    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" event={"ID":"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6","Type":"ContainerStarted","Data":"69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244"}
Nov 03 00:15:00.952227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.952082    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" event={"ID":"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6","Type":"ContainerStarted","Data":"7f0609531175ebbaca6fecc173e252d25b706bbb89c45c8b011a77b86a8a642b"}
Nov 03 00:15:00.995948 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:00.995891    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podStartSLOduration=1.995876745 podStartE2EDuration="1.995876745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:14:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:00.99435226 +0000 UTC m=+538.549704289" watchObservedRunningTime="2026-11-03 00:15:00.995876745 +0000 UTC m=+538.551228768"
Nov 03 00:15:01.000118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000088    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-certs\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000141    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-data\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000158    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-credential-socket\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000282    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-podinfo\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000313    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-socket\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000331    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-token\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000464    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-envoy\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000519 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000503    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gspwd\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-kube-api-access-gspwd\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.000606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.000590    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576d601b-8d25-454e-bbc4-e68a1d8483e4-istiod-ca-cert\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101421 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101384    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-data\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101421 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101423    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-credential-socket\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101446    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-podinfo\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101467    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-socket\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101484    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-token\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101556    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-envoy\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101597    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gspwd\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-kube-api-access-gspwd\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101677    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576d601b-8d25-454e-bbc4-e68a1d8483e4-istiod-ca-cert\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101735    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-certs\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101872    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-data\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.101966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101900    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-credential-socket\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.102173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.101982    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-socket\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.102173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.102104    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-certs\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.102795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.102772    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576d601b-8d25-454e-bbc4-e68a1d8483e4-istiod-ca-cert\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.104174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.104148    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-envoy\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.105039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.105019    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-podinfo\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.125088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.125018    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gspwd\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-kube-api-access-gspwd\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.128044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.127984    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-token\") pod \"e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") " pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.185390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.185357    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:01.510535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.510502    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"]
Nov 03 00:15:01.511993 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:01.511966    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod576d601b_8d25_454e_bbc4_e68a1d8483e4.slice/crio-0b1a9f00e8ff539344033a45d0781ef227f6fbf69b1ba0232b23c162dfc6f29d WatchSource:0}: Error finding container 0b1a9f00e8ff539344033a45d0781ef227f6fbf69b1ba0232b23c162dfc6f29d: Status 404 returned error can't find the container with id 0b1a9f00e8ff539344033a45d0781ef227f6fbf69b1ba0232b23c162dfc6f29d
Nov 03 00:15:01.514462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.514426    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:01.514547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.514517    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:01.514584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.514563    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:01.720324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.720297    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:01.725917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.725892    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:01.958771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.958739    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" event={"ID":"576d601b-8d25-454e-bbc4-e68a1d8483e4","Type":"ContainerStarted","Data":"2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371"}
Nov 03 00:15:01.959238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.958938    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:01.959238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.958999    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" event={"ID":"576d601b-8d25-454e-bbc4-e68a1d8483e4","Type":"ContainerStarted","Data":"0b1a9f00e8ff539344033a45d0781ef227f6fbf69b1ba0232b23c162dfc6f29d"}
Nov 03 00:15:01.959828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:01.959812    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:02.003771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:02.003713    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podStartSLOduration=2.003692443 podStartE2EDuration="2.003692443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:02.00298179 +0000 UTC m=+539.558333815" watchObservedRunningTime="2026-11-03 00:15:02.003692443 +0000 UTC m=+539.559044468"
Nov 03 00:15:02.186348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:02.186319    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:02.204727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:02.204684    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:02.963741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:02.963705    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:02.964731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:02.964713    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:03.423951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.423915    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"]
Nov 03 00:15:03.428676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.428655    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.448150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.448127    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-c600f063-serving-cert\""
Nov 03 00:15:03.449976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.449954    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"]
Nov 03 00:15:03.527448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.527419    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jx4fx\" (UniqueName: \"kubernetes.io/projected/eb91410e-134e-4b4f-9b9e-089f2d001ee4-kube-api-access-jx4fx\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.527622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.527549    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb91410e-134e-4b4f-9b9e-089f2d001ee4-openshift-service-ca\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.527622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.527613    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-maas-api-tls\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.527700 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.527637    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-metrics-certs\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.628470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.628444    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb91410e-134e-4b4f-9b9e-089f2d001ee4-openshift-service-ca\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.628652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.628512    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-maas-api-tls\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.628652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.628538    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-metrics-certs\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.628652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.628586    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jx4fx\" (UniqueName: \"kubernetes.io/projected/eb91410e-134e-4b4f-9b9e-089f2d001ee4-kube-api-access-jx4fx\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.629169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.629143    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb91410e-134e-4b4f-9b9e-089f2d001ee4-openshift-service-ca\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.631275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.631235    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-metrics-certs\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.631356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.631247    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-maas-api-tls\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.648138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.648107    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jx4fx\" (UniqueName: \"kubernetes.io/projected/eb91410e-134e-4b4f-9b9e-089f2d001ee4-kube-api-access-jx4fx\") pod \"maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:03.741473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:03.741385    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:04.044540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:04.044514    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"]
Nov 03 00:15:04.045690 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:04.045665    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb91410e_134e_4b4f_9b9e_089f2d001ee4.slice/crio-453bb8f806cc83097546c6fd69457a6c20077f3f96923cc11df85d9fb3cc3a5e WatchSource:0}: Error finding container 453bb8f806cc83097546c6fd69457a6c20077f3f96923cc11df85d9fb3cc3a5e: Status 404 returned error can't find the container with id 453bb8f806cc83097546c6fd69457a6c20077f3f96923cc11df85d9fb3cc3a5e
Nov 03 00:15:04.976604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:04.976563    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl" event={"ID":"eb91410e-134e-4b4f-9b9e-089f2d001ee4","Type":"ContainerStarted","Data":"da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8"}
Nov 03 00:15:04.976779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:04.976632    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl" event={"ID":"eb91410e-134e-4b4f-9b9e-089f2d001ee4","Type":"ContainerStarted","Data":"453bb8f806cc83097546c6fd69457a6c20077f3f96923cc11df85d9fb3cc3a5e"}
Nov 03 00:15:04.976862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:04.976838    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:05.057687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.057631    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl" podStartSLOduration=1.6400335670000001 podStartE2EDuration="2.057615523s" totalImagesPullingTime="417.581956ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:05.055698753 +0000 UTC m=+542.611050802" watchObservedRunningTime="2026-11-03 00:15:05.057615523 +0000 UTC m=+542.612967546"
Nov 03 00:15:05.592471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.592435    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"]
Nov 03 00:15:05.602423 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.602394    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.633019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.632991    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"]
Nov 03 00:15:05.643035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.643009    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-b4cc5943-serving-cert\""
Nov 03 00:15:05.751383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.751345    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-metrics-certs\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.751566 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.751422    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sprj\" (UniqueName: \"kubernetes.io/projected/6c63aec7-efd3-475c-9c53-f06d30ee22e9-kube-api-access-8sprj\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.751632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.751566    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-maas-api-tls\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.751697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.751678    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c63aec7-efd3-475c-9c53-f06d30ee22e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.852693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.852666    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sprj\" (UniqueName: \"kubernetes.io/projected/6c63aec7-efd3-475c-9c53-f06d30ee22e9-kube-api-access-8sprj\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.852897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.852724    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-maas-api-tls\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.852897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.852765    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c63aec7-efd3-475c-9c53-f06d30ee22e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.852897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.852816    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-metrics-certs\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.853406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.853377    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c63aec7-efd3-475c-9c53-f06d30ee22e9-openshift-service-ca\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.855514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.855490    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-metrics-certs\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.855607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.855549    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-maas-api-tls\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.875438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.875405    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sprj\" (UniqueName: \"kubernetes.io/projected/6c63aec7-efd3-475c-9c53-f06d30ee22e9-kube-api-access-8sprj\") pod \"maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.924647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.924609    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:05.925227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:05.925205    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:06.235408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:06.235379    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"]
Nov 03 00:15:06.236148 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:06.236122    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c63aec7_efd3_475c_9c53_f06d30ee22e9.slice/crio-198f63883a1910cdb702829fbb675c06452da9f0f3c193f1ecb6116122f4e273 WatchSource:0}: Error finding container 198f63883a1910cdb702829fbb675c06452da9f0f3c193f1ecb6116122f4e273: Status 404 returned error can't find the container with id 198f63883a1910cdb702829fbb675c06452da9f0f3c193f1ecb6116122f4e273
Nov 03 00:15:06.989619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:06.989580    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j" event={"ID":"6c63aec7-efd3-475c-9c53-f06d30ee22e9","Type":"ContainerStarted","Data":"6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8"}
Nov 03 00:15:06.989826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:06.989634    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j" event={"ID":"6c63aec7-efd3-475c-9c53-f06d30ee22e9","Type":"ContainerStarted","Data":"198f63883a1910cdb702829fbb675c06452da9f0f3c193f1ecb6116122f4e273"}
Nov 03 00:15:06.989826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:06.989786    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:07.027444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:07.027376    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j" podStartSLOduration=1.69539372 podStartE2EDuration="2.027356141s" totalImagesPullingTime="331.962421ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:07.026693502 +0000 UTC m=+544.582045719" watchObservedRunningTime="2026-11-03 00:15:07.027356141 +0000 UTC m=+544.582708169"
Nov 03 00:15:08.791240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.791203    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"]
Nov 03 00:15:08.794946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.794930    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:08.815221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.815192    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"]
Nov 03 00:15:08.857118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.857084    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-19590f89-dockercfg-fx6wm\""
Nov 03 00:15:08.857118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.857084    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-19590f89\""
Nov 03 00:15:08.890901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.890861    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd42448-6a2b-40e5-8c67-b5273bae7803-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:08.891104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.891078    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczkc\" (UniqueName: \"kubernetes.io/projected/6fd42448-6a2b-40e5-8c67-b5273bae7803-kube-api-access-vczkc\") pod \"payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:08.993267 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.993228    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd42448-6a2b-40e5-8c67-b5273bae7803-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:08.993408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.993338    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczkc\" (UniqueName: \"kubernetes.io/projected/6fd42448-6a2b-40e5-8c67-b5273bae7803-kube-api-access-vczkc\") pod \"payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:08.993910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:08.993892    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd42448-6a2b-40e5-8c67-b5273bae7803-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:09.017633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.017600    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczkc\" (UniqueName: \"kubernetes.io/projected/6fd42448-6a2b-40e5-8c67-b5273bae7803-kube-api-access-vczkc\") pod \"payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:09.109834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.109809    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:09.410912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.410888    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"]
Nov 03 00:15:09.412244 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:09.412218    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd42448_6a2b_40e5_8c67_b5273bae7803.slice/crio-b590602670efeca711b1e5d84608e334a225d445a1052a29ea47a6a54a7fe938 WatchSource:0}: Error finding container b590602670efeca711b1e5d84608e334a225d445a1052a29ea47a6a54a7fe938: Status 404 returned error can't find the container with id b590602670efeca711b1e5d84608e334a225d445a1052a29ea47a6a54a7fe938
Nov 03 00:15:09.422217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.422192    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"]
Nov 03 00:15:09.427280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.427236    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.444340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.444315    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-19590f89-serving-cert\""
Nov 03 00:15:09.445490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.445467    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"]
Nov 03 00:15:09.599552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.599506    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r7nl\" (UniqueName: \"kubernetes.io/projected/b71447ce-bbcb-49c4-9a74-54dc24973c37-kube-api-access-9r7nl\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.599708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.599567    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-maas-api-tls\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.599708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.599600    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-metrics-certs\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.599854 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.599825    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71447ce-bbcb-49c4-9a74-54dc24973c37-openshift-service-ca\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.701388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.701292    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71447ce-bbcb-49c4-9a74-54dc24973c37-openshift-service-ca\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.701568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.701430    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r7nl\" (UniqueName: \"kubernetes.io/projected/b71447ce-bbcb-49c4-9a74-54dc24973c37-kube-api-access-9r7nl\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.701568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.701458    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-maas-api-tls\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.701568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.701480    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-metrics-certs\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.702454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.702378    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71447ce-bbcb-49c4-9a74-54dc24973c37-openshift-service-ca\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.704177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.704153    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-maas-api-tls\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.704298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.704189    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-metrics-certs\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.719476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.719453    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r7nl\" (UniqueName: \"kubernetes.io/projected/b71447ce-bbcb-49c4-9a74-54dc24973c37-kube-api-access-9r7nl\") pod \"maas-api-e2e-ait-19590f89-7557694598-xhsm2\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:09.806692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:09.806663    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:10.006162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:10.006075    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5" event={"ID":"6fd42448-6a2b-40e5-8c67-b5273bae7803","Type":"ContainerStarted","Data":"4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d"}
Nov 03 00:15:10.006162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:10.006131    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5" event={"ID":"6fd42448-6a2b-40e5-8c67-b5273bae7803","Type":"ContainerStarted","Data":"b590602670efeca711b1e5d84608e334a225d445a1052a29ea47a6a54a7fe938"}
Nov 03 00:15:10.006375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:10.006198    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:10.070950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:10.070778    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5" podStartSLOduration=2.070758849 podStartE2EDuration="2.070758849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:10.069710601 +0000 UTC m=+547.625062625" watchObservedRunningTime="2026-11-03 00:15:10.070758849 +0000 UTC m=+547.626110874"
Nov 03 00:15:10.152970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:10.152931    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"]
Nov 03 00:15:10.153902 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:10.153873    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71447ce_bbcb_49c4_9a74_54dc24973c37.slice/crio-7069dd836d40d4b28416c00a625b1c4cfd8d5fc771143bb27096b1ff235d4db0 WatchSource:0}: Error finding container 7069dd836d40d4b28416c00a625b1c4cfd8d5fc771143bb27096b1ff235d4db0: Status 404 returned error can't find the container with id 7069dd836d40d4b28416c00a625b1c4cfd8d5fc771143bb27096b1ff235d4db0
Nov 03 00:15:10.987981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:10.987956    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:11.013287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:11.013233    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2" event={"ID":"b71447ce-bbcb-49c4-9a74-54dc24973c37","Type":"ContainerStarted","Data":"77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3"}
Nov 03 00:15:11.013287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:11.013294    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2" event={"ID":"b71447ce-bbcb-49c4-9a74-54dc24973c37","Type":"ContainerStarted","Data":"7069dd836d40d4b28416c00a625b1c4cfd8d5fc771143bb27096b1ff235d4db0"}
Nov 03 00:15:11.013533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:11.013458    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:11.069818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:11.069756    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2" podStartSLOduration=1.740685767 podStartE2EDuration="2.069738142s" totalImagesPullingTime="329.052375ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:11.067440352 +0000 UTC m=+548.622792378" watchObservedRunningTime="2026-11-03 00:15:11.069738142 +0000 UTC m=+548.625090170"
Nov 03 00:15:12.956727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:12.956637    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"]
Nov 03 00:15:12.963582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:12.963551    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:12.976994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:12.976957    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"]
Nov 03 00:15:12.986418 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:12.986392    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e3e6ed37-dockercfg-cx54q\""
Nov 03 00:15:12.986588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:12.986458    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e3e6ed37\""
Nov 03 00:15:13.000195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.000170    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:13.157050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.157012    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205d2a22-a46a-499c-b88e-6b658c56b368-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:13.157244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.157098    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcm9l\" (UniqueName: \"kubernetes.io/projected/205d2a22-a46a-499c-b88e-6b658c56b368-kube-api-access-zcm9l\") pod \"payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:13.258411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.258318    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205d2a22-a46a-499c-b88e-6b658c56b368-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:13.258411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.258381    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcm9l\" (UniqueName: \"kubernetes.io/projected/205d2a22-a46a-499c-b88e-6b658c56b368-kube-api-access-zcm9l\") pod \"payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:13.259034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.259013    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205d2a22-a46a-499c-b88e-6b658c56b368-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:13.277041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.277017    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcm9l\" (UniqueName: \"kubernetes.io/projected/205d2a22-a46a-499c-b88e-6b658c56b368-kube-api-access-zcm9l\") pod \"payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:13.277811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.277782    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:13.340729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.340698    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"]
Nov 03 00:15:13.350029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.350006    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.369839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.366229    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-a96a129b-serving-cert\""
Nov 03 00:15:13.370152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.370128    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"]
Nov 03 00:15:13.461169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.461141    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-maas-api-tls\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.461389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.461180    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-metrics-certs\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.461389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.461215    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cqlnw\" (UniqueName: \"kubernetes.io/projected/0a0fa9e1-c535-4905-8aba-6ec528695f51-kube-api-access-cqlnw\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.461467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.461401    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fa9e1-c535-4905-8aba-6ec528695f51-openshift-service-ca\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.562170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.562090    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cqlnw\" (UniqueName: \"kubernetes.io/projected/0a0fa9e1-c535-4905-8aba-6ec528695f51-kube-api-access-cqlnw\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.562170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.562151    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fa9e1-c535-4905-8aba-6ec528695f51-openshift-service-ca\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.562394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.562224    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-maas-api-tls\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.562394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.562247    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-metrics-certs\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.562851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.562821    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fa9e1-c535-4905-8aba-6ec528695f51-openshift-service-ca\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.564909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.564876    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-maas-api-tls\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.564996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.564953    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-metrics-certs\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.575549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.575527    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"]
Nov 03 00:15:13.577065 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:13.576966    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod205d2a22_a46a_499c_b88e_6b658c56b368.slice/crio-a961a50bf98b79bb5ffd7d2d1f2bf720faa2c23dbcd5c6f8a7004f9fbe3ff992 WatchSource:0}: Error finding container a961a50bf98b79bb5ffd7d2d1f2bf720faa2c23dbcd5c6f8a7004f9fbe3ff992: Status 404 returned error can't find the container with id a961a50bf98b79bb5ffd7d2d1f2bf720faa2c23dbcd5c6f8a7004f9fbe3ff992
Nov 03 00:15:13.593751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.593710    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cqlnw\" (UniqueName: \"kubernetes.io/projected/0a0fa9e1-c535-4905-8aba-6ec528695f51-kube-api-access-cqlnw\") pod \"maas-api-e2e-ait-a96a129b-fb7697488-bcxld\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.681413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.681389    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:13.859159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.859117    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:13.982157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:13.982134    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"]
Nov 03 00:15:13.983354 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:13.983316    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0fa9e1_c535_4905_8aba_6ec528695f51.slice/crio-8241426b7a22b0a8236af9d4296b34997dd7a0bc1a870f8af4cc62db8a89c44e WatchSource:0}: Error finding container 8241426b7a22b0a8236af9d4296b34997dd7a0bc1a870f8af4cc62db8a89c44e: Status 404 returned error can't find the container with id 8241426b7a22b0a8236af9d4296b34997dd7a0bc1a870f8af4cc62db8a89c44e
Nov 03 00:15:14.040996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:14.040954    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6" event={"ID":"205d2a22-a46a-499c-b88e-6b658c56b368","Type":"ContainerStarted","Data":"588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299"}
Nov 03 00:15:14.042928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:14.041034    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6" event={"ID":"205d2a22-a46a-499c-b88e-6b658c56b368","Type":"ContainerStarted","Data":"a961a50bf98b79bb5ffd7d2d1f2bf720faa2c23dbcd5c6f8a7004f9fbe3ff992"}
Nov 03 00:15:14.042928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:14.041509    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:14.045144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:14.044200    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld" event={"ID":"0a0fa9e1-c535-4905-8aba-6ec528695f51","Type":"ContainerStarted","Data":"8241426b7a22b0a8236af9d4296b34997dd7a0bc1a870f8af4cc62db8a89c44e"}
Nov 03 00:15:14.081336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:14.081246    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6" podStartSLOduration=2.081228436 podStartE2EDuration="2.081228436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:14.080547933 +0000 UTC m=+551.635899956" watchObservedRunningTime="2026-11-03 00:15:14.081228436 +0000 UTC m=+551.636580461"
Nov 03 00:15:15.050299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:15.050239    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld" event={"ID":"0a0fa9e1-c535-4905-8aba-6ec528695f51","Type":"ContainerStarted","Data":"37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be"}
Nov 03 00:15:15.050744 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:15.050426    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:15.098904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:15.098840    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld" podStartSLOduration=1.685879159 podStartE2EDuration="2.098818275s" totalImagesPullingTime="412.939116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:15.097661531 +0000 UTC m=+552.653013556" watchObservedRunningTime="2026-11-03 00:15:15.098818275 +0000 UTC m=+552.654170301"
Nov 03 00:15:17.035480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:17.035451    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:19.935283 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:19.934854    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:15:20.926952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.926868    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"]
Nov 03 00:15:20.931155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.931129    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:20.946949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.946916    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e3e6ed37-serving-cert\""
Nov 03 00:15:20.952967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.952942    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"]
Nov 03 00:15:20.982483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.982453    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6mz\" (UniqueName: \"kubernetes.io/projected/a7e21f59-6104-43d9-a6e7-862ed05c186b-kube-api-access-sn6mz\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:20.982644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.982621    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e21f59-6104-43d9-a6e7-862ed05c186b-openshift-service-ca\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:20.982704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.982685    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-maas-api-tls\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:20.982832 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:20.982817    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-metrics-certs\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.018319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.018289    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:21.065911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.063070    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:21.084185 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.084146    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-metrics-certs\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.084406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.084203    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6mz\" (UniqueName: \"kubernetes.io/projected/a7e21f59-6104-43d9-a6e7-862ed05c186b-kube-api-access-sn6mz\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.084406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.084384    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e21f59-6104-43d9-a6e7-862ed05c186b-openshift-service-ca\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.084547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.084435    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-maas-api-tls\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.086533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.086501    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e21f59-6104-43d9-a6e7-862ed05c186b-openshift-service-ca\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.106565 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.098231    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-maas-api-tls\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.124300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.122877    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-metrics-certs\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.130845 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.127444    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6mz\" (UniqueName: \"kubernetes.io/projected/a7e21f59-6104-43d9-a6e7-862ed05c186b-kube-api-access-sn6mz\") pod \"maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.249076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.248982    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:21.571980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.571948    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"]
Nov 03 00:15:21.572124 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:21.572024    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e21f59_6104_43d9_a6e7_862ed05c186b.slice/crio-2d31f428253f732cdbcd4a40e6d9cf9e9c6a4c0c511750b7dc543122061d0cbe WatchSource:0}: Error finding container 2d31f428253f732cdbcd4a40e6d9cf9e9c6a4c0c511750b7dc543122061d0cbe: Status 404 returned error can't find the container with id 2d31f428253f732cdbcd4a40e6d9cf9e9c6a4c0c511750b7dc543122061d0cbe
Nov 03 00:15:21.915317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:21.914289    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:22.127287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:22.127213    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz" event={"ID":"a7e21f59-6104-43d9-a6e7-862ed05c186b","Type":"ContainerStarted","Data":"8b995273805e3a07c0892cec319dacbb7587ccf80b7f4b9843f85c4029ca039b"}
Nov 03 00:15:22.127796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:22.127294    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz" event={"ID":"a7e21f59-6104-43d9-a6e7-862ed05c186b","Type":"ContainerStarted","Data":"2d31f428253f732cdbcd4a40e6d9cf9e9c6a4c0c511750b7dc543122061d0cbe"}
Nov 03 00:15:22.127796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:22.127325    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:22.164379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:22.164326    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz" podStartSLOduration=1.7867040950000002 podStartE2EDuration="2.164310574s" totalImagesPullingTime="377.606479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:22.164235477 +0000 UTC m=+559.719587502" watchObservedRunningTime="2026-11-03 00:15:22.164310574 +0000 UTC m=+559.719662597"
Nov 03 00:15:25.053907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:25.053873    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:25.165321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:25.164243    6000 generic.go:356] "Generic (PLEG): container finished" podID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerID="2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371" exitCode=0
Nov 03 00:15:25.165321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:25.164397    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" event={"ID":"576d601b-8d25-454e-bbc4-e68a1d8483e4","Type":"ContainerDied","Data":"2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371"}
Nov 03 00:15:25.165321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:25.164877    6000 scope.go:122] "RemoveContainer" containerID="2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371"
Nov 03 00:15:25.166484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:25.166447    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:25.166617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:25.166536    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:25.166617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:25.166586    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:26.172642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:26.172604    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" event={"ID":"576d601b-8d25-454e-bbc4-e68a1d8483e4","Type":"ContainerStarted","Data":"493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2"}
Nov 03 00:15:26.207285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:26.200389    6000 patch_prober.go:29] interesting pod/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 00:15:26.207285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:26.200476    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 00:15:26.217283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:26.216314    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:27.209282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:27.206650    6000 patch_prober.go:29] interesting pod/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 00:15:27.209282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:27.206712    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 00:15:28.143571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.143543    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:28.185290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.185242    6000 generic.go:356] "Generic (PLEG): container finished" podID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerID="69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244" exitCode=0
Nov 03 00:15:28.185459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.185317    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" event={"ID":"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6","Type":"ContainerDied","Data":"69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244"}
Nov 03 00:15:28.185684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.185659    6000 patch_prober.go:29] interesting pod/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 00:15:28.186007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.185706    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 00:15:28.186007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.185726    6000 scope.go:122] "RemoveContainer" containerID="69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244"
Nov 03 00:15:28.235379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.203052    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:28.235379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.203161    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:28.235379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:28.203220    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:29.190294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:29.187616    6000 patch_prober.go:29] interesting pod/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 00:15:29.190294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:29.187677    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 00:15:29.241279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:29.232350    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" event={"ID":"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6","Type":"ContainerStarted","Data":"96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e"}
Nov 03 00:15:29.346027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:29.345996    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:29.732279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:29.731268    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:29.732279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:29.731456    6000 patch_prober.go:29] interesting pod/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 00:15:29.732279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:29.731503    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 00:15:30.195282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:30.190669    6000 patch_prober.go:29] interesting pod/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 00:15:30.195282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:30.190732    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 00:15:30.721277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:30.720675    6000 patch_prober.go:29] interesting pod/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 00:15:30.721277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:30.720739    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 00:15:31.186023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:31.185992    6000 patch_prober.go:29] interesting pod/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Nov 03 00:15:31.186205 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:31.186046    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Nov 03 00:15:31.721322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:31.721283    6000 patch_prober.go:29] interesting pod/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 00:15:31.721837 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:31.721359    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 00:15:31.959248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:31.959213    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:32.211287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:32.210708    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:32.214320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:32.212303    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:32.214731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:32.214709    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:32.720987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:32.720956    6000 patch_prober.go:29] interesting pod/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 00:15:32.721181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:32.721011    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 00:15:33.720406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:33.720364    6000 patch_prober.go:29] interesting pod/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 00:15:33.720820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:33.720414    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 00:15:33.857320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:33.857287    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"]
Nov 03 00:15:33.857559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:33.857497    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl" podUID="eb91410e-134e-4b4f-9b9e-089f2d001ee4" containerName="maas-api" containerID="cri-o://da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8" gracePeriod=30
Nov 03 00:15:34.166900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.166874    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:34.265627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.265595    6000 generic.go:356] "Generic (PLEG): container finished" podID="eb91410e-134e-4b4f-9b9e-089f2d001ee4" containerID="da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8" exitCode=0
Nov 03 00:15:34.265844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.265664    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"
Nov 03 00:15:34.265844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.265720    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl" event={"ID":"eb91410e-134e-4b4f-9b9e-089f2d001ee4","Type":"ContainerDied","Data":"da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8"}
Nov 03 00:15:34.265844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.265760    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl" event={"ID":"eb91410e-134e-4b4f-9b9e-089f2d001ee4","Type":"ContainerDied","Data":"453bb8f806cc83097546c6fd69457a6c20077f3f96923cc11df85d9fb3cc3a5e"}
Nov 03 00:15:34.265844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.265779    6000 scope.go:122] "RemoveContainer" containerID="da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8"
Nov 03 00:15:34.275475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.275422    6000 scope.go:122] "RemoveContainer" containerID="da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8"
Nov 03 00:15:34.275765 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:34.275742    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8\": container with ID starting with da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8 not found: ID does not exist" containerID="da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8"
Nov 03 00:15:34.275866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.275777    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8"} err="failed to get container status \"da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8\": rpc error: code = NotFound desc = could not find container \"da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8\": container with ID starting with da6927a89443024134b9493b27cf2bd3fa74fc0cebd117796df57ae76fa059a8 not found: ID does not exist"
Nov 03 00:15:34.333339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.332593    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-maas-api-tls\") pod \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") "
Nov 03 00:15:34.333339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.332644    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb91410e-134e-4b4f-9b9e-089f2d001ee4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb91410e-134e-4b4f-9b9e-089f2d001ee4-openshift-service-ca\") pod \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") "
Nov 03 00:15:34.333339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.332744    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb91410e-134e-4b4f-9b9e-089f2d001ee4-kube-api-access-jx4fx\" (UniqueName: \"kubernetes.io/projected/eb91410e-134e-4b4f-9b9e-089f2d001ee4-kube-api-access-jx4fx\") pod \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") "
Nov 03 00:15:34.333339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.332844    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-metrics-certs\") pod \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\" (UID: \"eb91410e-134e-4b4f-9b9e-089f2d001ee4\") "
Nov 03 00:15:34.345043 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.334402    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb91410e-134e-4b4f-9b9e-089f2d001ee4-openshift-service-ca" pod "eb91410e-134e-4b4f-9b9e-089f2d001ee4" (UID: "eb91410e-134e-4b4f-9b9e-089f2d001ee4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:34.352856 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.352713    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-maas-api-tls" pod "eb91410e-134e-4b4f-9b9e-089f2d001ee4" (UID: "eb91410e-134e-4b4f-9b9e-089f2d001ee4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:34.353217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.353173    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb91410e-134e-4b4f-9b9e-089f2d001ee4-kube-api-access-jx4fx" pod "eb91410e-134e-4b4f-9b9e-089f2d001ee4" (UID: "eb91410e-134e-4b4f-9b9e-089f2d001ee4"). InnerVolumeSpecName "kube-api-access-jx4fx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:34.353517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.353485    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-metrics-certs" pod "eb91410e-134e-4b4f-9b9e-089f2d001ee4" (UID: "eb91410e-134e-4b4f-9b9e-089f2d001ee4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:34.434514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.434484    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:34.434514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.434511    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb91410e-134e-4b4f-9b9e-089f2d001ee4-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:34.434514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.434521    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jx4fx\" (UniqueName: \"kubernetes.io/projected/eb91410e-134e-4b4f-9b9e-089f2d001ee4-kube-api-access-jx4fx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:34.434761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.434532    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb91410e-134e-4b4f-9b9e-089f2d001ee4-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:34.624907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.624868    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"]
Nov 03 00:15:34.637548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.637517    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-c600f063-7bcd788c8c-mkfcl"]
Nov 03 00:15:34.720742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.720699    6000 patch_prober.go:29] interesting pod/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 03 00:15:34.721157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:34.720783    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 03 00:15:35.297472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:35.297432    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"]
Nov 03 00:15:35.724942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:35.724917    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:35.726108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:35.726091    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:36.211201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.211169    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"]
Nov 03 00:15:36.211419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.211393    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j" podUID="6c63aec7-efd3-475c-9c53-f06d30ee22e9" containerName="maas-api" containerID="cri-o://6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8" gracePeriod=30
Nov 03 00:15:36.493833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.493803    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:36.565950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.565921    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb91410e-134e-4b4f-9b9e-089f2d001ee4" path="/var/lib/kubelet/pods/eb91410e-134e-4b4f-9b9e-089f2d001ee4/volumes"
Nov 03 00:15:36.663965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.663900    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-maas-api-tls\") pod \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") "
Nov 03 00:15:36.664165 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.663989    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-metrics-certs\") pod \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") "
Nov 03 00:15:36.664165 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.664055    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c63aec7-efd3-475c-9c53-f06d30ee22e9-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c63aec7-efd3-475c-9c53-f06d30ee22e9-openshift-service-ca\") pod \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") "
Nov 03 00:15:36.664165 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.664103    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c63aec7-efd3-475c-9c53-f06d30ee22e9-kube-api-access-8sprj\" (UniqueName: \"kubernetes.io/projected/6c63aec7-efd3-475c-9c53-f06d30ee22e9-kube-api-access-8sprj\") pod \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\" (UID: \"6c63aec7-efd3-475c-9c53-f06d30ee22e9\") "
Nov 03 00:15:36.664772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.664712    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c63aec7-efd3-475c-9c53-f06d30ee22e9-openshift-service-ca" pod "6c63aec7-efd3-475c-9c53-f06d30ee22e9" (UID: "6c63aec7-efd3-475c-9c53-f06d30ee22e9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:36.666732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.666706    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-maas-api-tls" pod "6c63aec7-efd3-475c-9c53-f06d30ee22e9" (UID: "6c63aec7-efd3-475c-9c53-f06d30ee22e9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:36.666857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.666759    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-metrics-certs" pod "6c63aec7-efd3-475c-9c53-f06d30ee22e9" (UID: "6c63aec7-efd3-475c-9c53-f06d30ee22e9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:36.667019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.666996    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c63aec7-efd3-475c-9c53-f06d30ee22e9-kube-api-access-8sprj" pod "6c63aec7-efd3-475c-9c53-f06d30ee22e9" (UID: "6c63aec7-efd3-475c-9c53-f06d30ee22e9"). InnerVolumeSpecName "kube-api-access-8sprj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:36.766004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.765915    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:36.766004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.765942    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c63aec7-efd3-475c-9c53-f06d30ee22e9-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:36.766004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.765952    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c63aec7-efd3-475c-9c53-f06d30ee22e9-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:36.766004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:36.765961    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sprj\" (UniqueName: \"kubernetes.io/projected/6c63aec7-efd3-475c-9c53-f06d30ee22e9-kube-api-access-8sprj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:37.283741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.283710    6000 generic.go:356] "Generic (PLEG): container finished" podID="6c63aec7-efd3-475c-9c53-f06d30ee22e9" containerID="6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8" exitCode=0
Nov 03 00:15:37.283931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.283765    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"
Nov 03 00:15:37.283931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.283794    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j" event={"ID":"6c63aec7-efd3-475c-9c53-f06d30ee22e9","Type":"ContainerDied","Data":"6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8"}
Nov 03 00:15:37.283931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.283855    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j" event={"ID":"6c63aec7-efd3-475c-9c53-f06d30ee22e9","Type":"ContainerDied","Data":"198f63883a1910cdb702829fbb675c06452da9f0f3c193f1ecb6116122f4e273"}
Nov 03 00:15:37.283931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.283885    6000 scope.go:122] "RemoveContainer" containerID="6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8"
Nov 03 00:15:37.294592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.294562    6000 scope.go:122] "RemoveContainer" containerID="6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8"
Nov 03 00:15:37.300371 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:37.300338    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8\": container with ID starting with 6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8 not found: ID does not exist" containerID="6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8"
Nov 03 00:15:37.300493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.300386    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8"} err="failed to get container status \"6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8\": rpc error: code = NotFound desc = could not find container \"6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8\": container with ID starting with 6f7a2e06235c4febad10314f513b02137c549301522f47deb5ef99aba6cd56d8 not found: ID does not exist"
Nov 03 00:15:37.335171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.335144    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"]
Nov 03 00:15:37.346666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:37.346632    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-b4cc5943-5c78ff95cf-5vj5j"]
Nov 03 00:15:38.314827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:38.314789    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:38.522534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:38.522504    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"]
Nov 03 00:15:38.522807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:38.522754    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2" podUID="b71447ce-bbcb-49c4-9a74-54dc24973c37" containerName="maas-api" containerID="cri-o://77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3" gracePeriod=30
Nov 03 00:15:38.569827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:38.569739    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c63aec7-efd3-475c-9c53-f06d30ee22e9" path="/var/lib/kubelet/pods/6c63aec7-efd3-475c-9c53-f06d30ee22e9/volumes"
Nov 03 00:15:38.825545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:38.825468    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:38.900699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:38.900658    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b71447ce-bbcb-49c4-9a74-54dc24973c37-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71447ce-bbcb-49c4-9a74-54dc24973c37-openshift-service-ca\") pod \"b71447ce-bbcb-49c4-9a74-54dc24973c37\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") "
Nov 03 00:15:38.901012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:38.900991    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b71447ce-bbcb-49c4-9a74-54dc24973c37-openshift-service-ca" pod "b71447ce-bbcb-49c4-9a74-54dc24973c37" (UID: "b71447ce-bbcb-49c4-9a74-54dc24973c37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:39.001563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.001524    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-metrics-certs\") pod \"b71447ce-bbcb-49c4-9a74-54dc24973c37\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") "
Nov 03 00:15:39.001781 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.001606    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b71447ce-bbcb-49c4-9a74-54dc24973c37-kube-api-access-9r7nl\" (UniqueName: \"kubernetes.io/projected/b71447ce-bbcb-49c4-9a74-54dc24973c37-kube-api-access-9r7nl\") pod \"b71447ce-bbcb-49c4-9a74-54dc24973c37\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") "
Nov 03 00:15:39.001781 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.001659    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-maas-api-tls\") pod \"b71447ce-bbcb-49c4-9a74-54dc24973c37\" (UID: \"b71447ce-bbcb-49c4-9a74-54dc24973c37\") "
Nov 03 00:15:39.004050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.004021    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b71447ce-bbcb-49c4-9a74-54dc24973c37-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:39.006415 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.006365    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-maas-api-tls" pod "b71447ce-bbcb-49c4-9a74-54dc24973c37" (UID: "b71447ce-bbcb-49c4-9a74-54dc24973c37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:39.025300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.014966    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b71447ce-bbcb-49c4-9a74-54dc24973c37-kube-api-access-9r7nl" pod "b71447ce-bbcb-49c4-9a74-54dc24973c37" (UID: "b71447ce-bbcb-49c4-9a74-54dc24973c37"). InnerVolumeSpecName "kube-api-access-9r7nl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:39.025300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.015061    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-metrics-certs" pod "b71447ce-bbcb-49c4-9a74-54dc24973c37" (UID: "b71447ce-bbcb-49c4-9a74-54dc24973c37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:39.104974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.104938    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:39.104974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.104966    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r7nl\" (UniqueName: \"kubernetes.io/projected/b71447ce-bbcb-49c4-9a74-54dc24973c37-kube-api-access-9r7nl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:39.104974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.104975    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b71447ce-bbcb-49c4-9a74-54dc24973c37-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:39.296838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.296799    6000 generic.go:356] "Generic (PLEG): container finished" podID="b71447ce-bbcb-49c4-9a74-54dc24973c37" containerID="77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3" exitCode=0
Nov 03 00:15:39.297013 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.296834    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2" event={"ID":"b71447ce-bbcb-49c4-9a74-54dc24973c37","Type":"ContainerDied","Data":"77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3"}
Nov 03 00:15:39.297013 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.296891    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2" event={"ID":"b71447ce-bbcb-49c4-9a74-54dc24973c37","Type":"ContainerDied","Data":"7069dd836d40d4b28416c00a625b1c4cfd8d5fc771143bb27096b1ff235d4db0"}
Nov 03 00:15:39.297013 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.296916    6000 scope.go:122] "RemoveContainer" containerID="77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3"
Nov 03 00:15:39.297013 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.296851    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"
Nov 03 00:15:39.307236 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.307219    6000 scope.go:122] "RemoveContainer" containerID="77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3"
Nov 03 00:15:39.307513 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:39.307495    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3\": container with ID starting with 77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3 not found: ID does not exist" containerID="77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3"
Nov 03 00:15:39.307561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.307522    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3"} err="failed to get container status \"77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3\": rpc error: code = NotFound desc = could not find container \"77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3\": container with ID starting with 77ade388808926b71f4e3bff41965d25edbb72d8b6f204aa423fc255fc612ef3 not found: ID does not exist"
Nov 03 00:15:39.346096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.346067    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"]
Nov 03 00:15:39.365340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.364135    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-19590f89-7557694598-xhsm2"]
Nov 03 00:15:39.945603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.945564    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"]
Nov 03 00:15:39.945841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:39.945814    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy" containerID="cri-o://493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2" gracePeriod=30
Nov 03 00:15:40.298537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.298439    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5" podUID="6fd42448-6a2b-40e5-8c67-b5273bae7803" containerName="payload-pre-processing" containerID="cri-o://4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d" gracePeriod=25
Nov 03 00:15:40.547805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.547780    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:40.566499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.566434    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b71447ce-bbcb-49c4-9a74-54dc24973c37" path="/var/lib/kubelet/pods/b71447ce-bbcb-49c4-9a74-54dc24973c37/volumes"
Nov 03 00:15:40.621194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.621162    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fd42448-6a2b-40e5-8c67-b5273bae7803-kube-api-access-vczkc\" (UniqueName: \"kubernetes.io/projected/6fd42448-6a2b-40e5-8c67-b5273bae7803-kube-api-access-vczkc\") pod \"6fd42448-6a2b-40e5-8c67-b5273bae7803\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") "
Nov 03 00:15:40.621383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.621305    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fd42448-6a2b-40e5-8c67-b5273bae7803-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd42448-6a2b-40e5-8c67-b5273bae7803-plugins-config-volume\") pod \"6fd42448-6a2b-40e5-8c67-b5273bae7803\" (UID: \"6fd42448-6a2b-40e5-8c67-b5273bae7803\") "
Nov 03 00:15:40.621761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.621732    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fd42448-6a2b-40e5-8c67-b5273bae7803-plugins-config-volume" pod "6fd42448-6a2b-40e5-8c67-b5273bae7803" (UID: "6fd42448-6a2b-40e5-8c67-b5273bae7803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:40.623435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.623410    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fd42448-6a2b-40e5-8c67-b5273bae7803-kube-api-access-vczkc" pod "6fd42448-6a2b-40e5-8c67-b5273bae7803" (UID: "6fd42448-6a2b-40e5-8c67-b5273bae7803"). InnerVolumeSpecName "kube-api-access-vczkc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:40.722563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.722530    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fd42448-6a2b-40e5-8c67-b5273bae7803-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:40.722563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.722562    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vczkc\" (UniqueName: \"kubernetes.io/projected/6fd42448-6a2b-40e5-8c67-b5273bae7803-kube-api-access-vczkc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:40.828006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.827920    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"]
Nov 03 00:15:40.828199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:40.828173    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld" podUID="0a0fa9e1-c535-4905-8aba-6ec528695f51" containerName="maas-api" containerID="cri-o://37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be" gracePeriod=30
Nov 03 00:15:41.115521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.115493    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:41.126114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.126086    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0fa9e1-c535-4905-8aba-6ec528695f51-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fa9e1-c535-4905-8aba-6ec528695f51-openshift-service-ca\") pod \"0a0fa9e1-c535-4905-8aba-6ec528695f51\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") "
Nov 03 00:15:41.126299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.126211    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-metrics-certs\") pod \"0a0fa9e1-c535-4905-8aba-6ec528695f51\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") "
Nov 03 00:15:41.126382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.126319    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-maas-api-tls\") pod \"0a0fa9e1-c535-4905-8aba-6ec528695f51\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") "
Nov 03 00:15:41.126382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.126343    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0fa9e1-c535-4905-8aba-6ec528695f51-kube-api-access-cqlnw\" (UniqueName: \"kubernetes.io/projected/0a0fa9e1-c535-4905-8aba-6ec528695f51-kube-api-access-cqlnw\") pod \"0a0fa9e1-c535-4905-8aba-6ec528695f51\" (UID: \"0a0fa9e1-c535-4905-8aba-6ec528695f51\") "
Nov 03 00:15:41.126490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.126413    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0fa9e1-c535-4905-8aba-6ec528695f51-openshift-service-ca" pod "0a0fa9e1-c535-4905-8aba-6ec528695f51" (UID: "0a0fa9e1-c535-4905-8aba-6ec528695f51"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:41.126659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.126641    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fa9e1-c535-4905-8aba-6ec528695f51-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.128607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.128578    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-metrics-certs" pod "0a0fa9e1-c535-4905-8aba-6ec528695f51" (UID: "0a0fa9e1-c535-4905-8aba-6ec528695f51"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:41.128722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.128692    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0fa9e1-c535-4905-8aba-6ec528695f51-kube-api-access-cqlnw" pod "0a0fa9e1-c535-4905-8aba-6ec528695f51" (UID: "0a0fa9e1-c535-4905-8aba-6ec528695f51"). InnerVolumeSpecName "kube-api-access-cqlnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:41.128777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.128731    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-maas-api-tls" pod "0a0fa9e1-c535-4905-8aba-6ec528695f51" (UID: "0a0fa9e1-c535-4905-8aba-6ec528695f51"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:41.192098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192055    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"]
Nov 03 00:15:41.192580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192566    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0a0fa9e1-c535-4905-8aba-6ec528695f51" containerName="maas-api"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192582    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0fa9e1-c535-4905-8aba-6ec528695f51" containerName="maas-api"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192603    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6fd42448-6a2b-40e5-8c67-b5273bae7803" containerName="payload-pre-processing"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192609    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fd42448-6a2b-40e5-8c67-b5273bae7803" containerName="payload-pre-processing"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192617    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b71447ce-bbcb-49c4-9a74-54dc24973c37" containerName="maas-api"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192621    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b71447ce-bbcb-49c4-9a74-54dc24973c37" containerName="maas-api"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192632    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6c63aec7-efd3-475c-9c53-f06d30ee22e9" containerName="maas-api"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192637    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c63aec7-efd3-475c-9c53-f06d30ee22e9" containerName="maas-api"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192655    6000 cpu_manager.go:414] "containerMap: removing container" podUID="eb91410e-134e-4b4f-9b9e-089f2d001ee4" containerName="maas-api"
Nov 03 00:15:41.192664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192663    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb91410e-134e-4b4f-9b9e-089f2d001ee4" containerName="maas-api"
Nov 03 00:15:41.192994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192735    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb91410e-134e-4b4f-9b9e-089f2d001ee4" containerName="maas-api"
Nov 03 00:15:41.192994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192752    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0fa9e1-c535-4905-8aba-6ec528695f51" containerName="maas-api"
Nov 03 00:15:41.192994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192760    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fd42448-6a2b-40e5-8c67-b5273bae7803" containerName="payload-pre-processing"
Nov 03 00:15:41.192994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192766    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b71447ce-bbcb-49c4-9a74-54dc24973c37" containerName="maas-api"
Nov 03 00:15:41.192994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.192772    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c63aec7-efd3-475c-9c53-f06d30ee22e9" containerName="maas-api"
Nov 03 00:15:41.195206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.195187    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.209235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.209211    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fc9316-openshift-default-dockercfg-8zk4q\""
Nov 03 00:15:41.224853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.224826    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"]
Nov 03 00:15:41.227356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227334    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm6qk\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-kube-api-access-gm6qk\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227365    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227510    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227552    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227613    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227696    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227728    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227857    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227894    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.227975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227963    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.228007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227983    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cqlnw\" (UniqueName: \"kubernetes.io/projected/0a0fa9e1-c535-4905-8aba-6ec528695f51-kube-api-access-cqlnw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.228007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.227995    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fa9e1-c535-4905-8aba-6ec528695f51-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:41.311543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.311503    6000 generic.go:356] "Generic (PLEG): container finished" podID="6fd42448-6a2b-40e5-8c67-b5273bae7803" containerID="4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d" exitCode=0
Nov 03 00:15:41.311708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.311536    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5" event={"ID":"6fd42448-6a2b-40e5-8c67-b5273bae7803","Type":"ContainerDied","Data":"4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d"}
Nov 03 00:15:41.311708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.311559    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"
Nov 03 00:15:41.311708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.311604    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5" event={"ID":"6fd42448-6a2b-40e5-8c67-b5273bae7803","Type":"ContainerDied","Data":"b590602670efeca711b1e5d84608e334a225d445a1052a29ea47a6a54a7fe938"}
Nov 03 00:15:41.311708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.311634    6000 scope.go:122] "RemoveContainer" containerID="4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d"
Nov 03 00:15:41.313520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.313278    6000 generic.go:356] "Generic (PLEG): container finished" podID="0a0fa9e1-c535-4905-8aba-6ec528695f51" containerID="37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be" exitCode=0
Nov 03 00:15:41.313520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.313312    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld" event={"ID":"0a0fa9e1-c535-4905-8aba-6ec528695f51","Type":"ContainerDied","Data":"37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be"}
Nov 03 00:15:41.313520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.313346    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld" event={"ID":"0a0fa9e1-c535-4905-8aba-6ec528695f51","Type":"ContainerDied","Data":"8241426b7a22b0a8236af9d4296b34997dd7a0bc1a870f8af4cc62db8a89c44e"}
Nov 03 00:15:41.313520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.313390    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"
Nov 03 00:15:41.328456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328414    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm6qk\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-kube-api-access-gm6qk\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328452    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328501    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328528    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328545    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328603    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328666    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328770    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.328841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.328804    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.329120 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.329002    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.329120 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.329098    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.329223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.329177    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.329372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.329326    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.332609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.330819    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.332609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.331919    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.332609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.331669    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.332840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.332804    6000 scope.go:122] "RemoveContainer" containerID="4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d"
Nov 03 00:15:41.333183 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:41.333162    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d\": container with ID starting with 4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d not found: ID does not exist" containerID="4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d"
Nov 03 00:15:41.333247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.333189    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d"} err="failed to get container status \"4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d\": rpc error: code = NotFound desc = could not find container \"4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d\": container with ID starting with 4b394d9e27b80eb8a46a7ee7a1d03a1cf4c38e771327c7df07c745a2f6b9752d not found: ID does not exist"
Nov 03 00:15:41.333247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.333207    6000 scope.go:122] "RemoveContainer" containerID="37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be"
Nov 03 00:15:41.354446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.354428    6000 scope.go:122] "RemoveContainer" containerID="37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be"
Nov 03 00:15:41.354822 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:41.354794    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be\": container with ID starting with 37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be not found: ID does not exist" containerID="37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be"
Nov 03 00:15:41.354898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.354828    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be"} err="failed to get container status \"37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be\": rpc error: code = NotFound desc = could not find container \"37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be\": container with ID starting with 37d62535a4714a9b9099d83e2dde0ef6ff7dad6123247c324b5a0c8a6ff447be not found: ID does not exist"
Nov 03 00:15:41.354898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.354867    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm6qk\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-kube-api-access-gm6qk\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.355038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.355021    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.389126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.389051    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"]
Nov 03 00:15:41.396838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.396815    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19590f89-75b6594799-w8kj5"]
Nov 03 00:15:41.429883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.429853    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"]
Nov 03 00:15:41.433838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.433816    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld"]
Nov 03 00:15:41.518875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.516863    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:41.854127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.854101    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"]
Nov 03 00:15:41.857236 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:41.857210    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e4dfcbe_7183_4a23_87b7_8da7f51823a7.slice/crio-fc1b222f7f9e4d1bb90d9ddffeb32eb14a8a5dfeb3ee46132461e47790b1d9c7 WatchSource:0}: Error finding container fc1b222f7f9e4d1bb90d9ddffeb32eb14a8a5dfeb3ee46132461e47790b1d9c7: Status 404 returned error can't find the container with id fc1b222f7f9e4d1bb90d9ddffeb32eb14a8a5dfeb3ee46132461e47790b1d9c7
Nov 03 00:15:41.859874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.859840    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:41.859980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.859912    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:41.859980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.859945    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:41.980119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.980086    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"]
Nov 03 00:15:41.980378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:41.980354    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy" containerID="cri-o://96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e" gracePeriod=30
Nov 03 00:15:42.323869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:42.323791    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerStarted","Data":"4960a94afeb68a0a51ae3656361643a1caff85e8a44e6dd6ccd2de3db27290c0"}
Nov 03 00:15:42.323869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:42.323846    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerStarted","Data":"fc1b222f7f9e4d1bb90d9ddffeb32eb14a8a5dfeb3ee46132461e47790b1d9c7"}
Nov 03 00:15:42.372529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:42.372467    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podStartSLOduration=1.372449979 podStartE2EDuration="1.372449979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:42.371328945 +0000 UTC m=+579.926680968" watchObservedRunningTime="2026-11-03 00:15:42.372449979 +0000 UTC m=+579.927802004"
Nov 03 00:15:42.518979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:42.518950    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:42.523724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:42.523697    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:42.571581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:42.571545    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0fa9e1-c535-4905-8aba-6ec528695f51" path="/var/lib/kubelet/pods/0a0fa9e1-c535-4905-8aba-6ec528695f51/volumes"
Nov 03 00:15:42.572114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:42.572096    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fd42448-6a2b-40e5-8c67-b5273bae7803" path="/var/lib/kubelet/pods/6fd42448-6a2b-40e5-8c67-b5273bae7803/volumes"
Nov 03 00:15:43.261243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.261209    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"]
Nov 03 00:15:43.266487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.266465    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.286762 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.285443    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"]
Nov 03 00:15:43.330051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.330023    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:43.330939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.330921    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:15:43.348093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348066    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348095    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348119    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348358    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348418 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348405    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348543    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348583    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348612    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kd86\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-kube-api-access-4kd86\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.348755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.348739    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.350170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.350154    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-4c61c8-openshift-default-dockercfg-sclgr\""
Nov 03 00:15:43.449669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.449630    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.449859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.449694    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.449859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.449743    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.449859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.449777    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.449859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.449822    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kd86\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-kube-api-access-4kd86\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.449901    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.449973    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.450001    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.450038    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.450428    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.450677    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.450797    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.450953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.450855    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.451061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.451041    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.452783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.452761    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.453586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.453562    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.473759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.473730    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.480747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.480722    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kd86\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-kube-api-access-4kd86\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.586709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.586630    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:43.895880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.895854    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"]
Nov 03 00:15:43.897591 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:43.897554    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d39152_3fe9_47bd_b166_9a84ac7a5f98.slice/crio-c3b296fbbadba7c0b1c7e710db3630781fed7f012baf85d376ae31e197982658 WatchSource:0}: Error finding container c3b296fbbadba7c0b1c7e710db3630781fed7f012baf85d376ae31e197982658: Status 404 returned error can't find the container with id c3b296fbbadba7c0b1c7e710db3630781fed7f012baf85d376ae31e197982658
Nov 03 00:15:43.900522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.900490    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:43.900623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.900554    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:43.900623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:43.900586    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:15:44.061220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:44.061179    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ait-a96a129b-fb7697488-bcxld" podUID="0a0fa9e1-c535-4905-8aba-6ec528695f51" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.48:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:15:44.336759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:44.336657    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerStarted","Data":"e96de02f82f86a34da6b11d24e1416e2e1688b00ef11463548fe006b3a003032"}
Nov 03 00:15:44.336759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:44.336717    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerStarted","Data":"c3b296fbbadba7c0b1c7e710db3630781fed7f012baf85d376ae31e197982658"}
Nov 03 00:15:44.379641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:44.379590    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podStartSLOduration=1.3795763509999999 podStartE2EDuration="1.379576351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:44.37718247 +0000 UTC m=+581.932534494" watchObservedRunningTime="2026-11-03 00:15:44.379576351 +0000 UTC m=+581.934928375"
Nov 03 00:15:44.587592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:44.587489    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:44.594011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:44.593980    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:45.231798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.231774    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:45.277214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.276993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-kube-api-access-gspwd\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-kube-api-access-gspwd\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277462    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-envoy\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277524    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/576d601b-8d25-454e-bbc4-e68a1d8483e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576d601b-8d25-454e-bbc4-e68a1d8483e4-istiod-ca-cert\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277572    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-data\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277614    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-podinfo\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277678    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-certs\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277712    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-socket\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277776    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-token\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-token\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.277941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.277873    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-credential-socket\") pod \"576d601b-8d25-454e-bbc4-e68a1d8483e4\" (UID: \"576d601b-8d25-454e-bbc4-e68a1d8483e4\") "
Nov 03 00:15:45.278161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.278136    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/576d601b-8d25-454e-bbc4-e68a1d8483e4-istiod-ca-cert" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:45.278342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.278322    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/576d601b-8d25-454e-bbc4-e68a1d8483e4-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.278478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.278395    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-credential-socket" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:45.279716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.278622    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-certs" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:45.279716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.278702    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-data" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:45.279716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.278914    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-socket" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:45.289289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.283384    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-podinfo" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:15:45.289289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.283396    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-envoy" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:45.289289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.283424    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-token" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:45.289289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.283432    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-kube-api-access-gspwd" pod "576d601b-8d25-454e-bbc4-e68a1d8483e4" (UID: "576d601b-8d25-454e-bbc4-e68a1d8483e4"). InnerVolumeSpecName "kube-api-access-gspwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:45.344166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.344135    6000 generic.go:356] "Generic (PLEG): container finished" podID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerID="493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2" exitCode=0
Nov 03 00:15:45.344617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.344193    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"
Nov 03 00:15:45.344617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.344227    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" event={"ID":"576d601b-8d25-454e-bbc4-e68a1d8483e4","Type":"ContainerDied","Data":"493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2"}
Nov 03 00:15:45.344617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.344309    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk" event={"ID":"576d601b-8d25-454e-bbc4-e68a1d8483e4","Type":"ContainerDied","Data":"0b1a9f00e8ff539344033a45d0781ef227f6fbf69b1ba0232b23c162dfc6f29d"}
Nov 03 00:15:45.344617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.344336    6000 scope.go:122] "RemoveContainer" containerID="493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2"
Nov 03 00:15:45.344933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.344914    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:45.345756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.345724    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:15:45.354355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.354338    6000 scope.go:122] "RemoveContainer" containerID="2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371"
Nov 03 00:15:45.362401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.362383    6000 scope.go:122] "RemoveContainer" containerID="493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2"
Nov 03 00:15:45.362651 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:45.362632    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2\": container with ID starting with 493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2 not found: ID does not exist" containerID="493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2"
Nov 03 00:15:45.362693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.362660    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2"} err="failed to get container status \"493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2\": rpc error: code = NotFound desc = could not find container \"493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2\": container with ID starting with 493b38fb5a47c59480da1fde15ff70e12f8cf842012e9272f9199a0b722f34a2 not found: ID does not exist"
Nov 03 00:15:45.362693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.362678    6000 scope.go:122] "RemoveContainer" containerID="2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371"
Nov 03 00:15:45.362885 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:45.362864    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371\": container with ID starting with 2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371 not found: ID does not exist" containerID="2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371"
Nov 03 00:15:45.362922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.362892    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371"} err="failed to get container status \"2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371\": rpc error: code = NotFound desc = could not find container \"2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371\": container with ID starting with 2d8b887871089aac79a2b20f9562cbcdcd684e8641129c7be8e6a59eb6bb9371 not found: ID does not exist"
Nov 03 00:15:45.379993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.379958    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.380093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.380001    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.380093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.380017    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.380093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.380039    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.380093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.380054    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.380093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.380068    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.380093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.380083    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/576d601b-8d25-454e-bbc4-e68a1d8483e4-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.380321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.380099    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gspwd\" (UniqueName: \"kubernetes.io/projected/576d601b-8d25-454e-bbc4-e68a1d8483e4-kube-api-access-gspwd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:45.393000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.392965    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"]
Nov 03 00:15:45.399097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.399075    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-19590f89-openshift-default-6fc56f545f-ccltk"]
Nov 03 00:15:45.606136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.606103    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"]
Nov 03 00:15:45.606583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.606569    6000 cpu_manager.go:414] "containerMap: removing container" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy"
Nov 03 00:15:45.606583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.606583    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy"
Nov 03 00:15:45.606753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.606691    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy"
Nov 03 00:15:45.606753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.606702    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy"
Nov 03 00:15:45.609753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.609734    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.629097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.629077    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-99bdb8-serving-cert\""
Nov 03 00:15:45.632051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.632025    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"]
Nov 03 00:15:45.684170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.684139    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-metrics-certs\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.684362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.684179    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.684362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.684206    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de14e2d3-979f-4186-91fe-6f8630291d8a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.684362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.684353    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqcnw\" (UniqueName: \"kubernetes.io/projected/de14e2d3-979f-4186-91fe-6f8630291d8a-kube-api-access-nqcnw\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.785788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.785756    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-metrics-certs\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.785991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.785802    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.785991 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:45.785918    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-serving-cert: secret "maas-api-e2e-worker-w4-99bdb8-serving-cert" not found
Nov 03 00:15:45.785991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.785930    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de14e2d3-979f-4186-91fe-6f8630291d8a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.786152 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:45.786000    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls podName:de14e2d3-979f-4186-91fe-6f8630291d8a nodeName:}" failed. No retries permitted until 2026-11-03 00:15:46.285978458 +0000 UTC m=+583.841330479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls") pod "maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n" (UID: "de14e2d3-979f-4186-91fe-6f8630291d8a") : secret "maas-api-e2e-worker-w4-99bdb8-serving-cert" not found
Nov 03 00:15:45.786152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.786074    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqcnw\" (UniqueName: \"kubernetes.io/projected/de14e2d3-979f-4186-91fe-6f8630291d8a-kube-api-access-nqcnw\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.786652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.786634    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de14e2d3-979f-4186-91fe-6f8630291d8a-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.788357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.788337    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-metrics-certs\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:45.810495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:45.810473    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqcnw\" (UniqueName: \"kubernetes.io/projected/de14e2d3-979f-4186-91fe-6f8630291d8a-kube-api-access-nqcnw\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:46.293084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:46.293048    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:46.295586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:46.295560    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls\") pod \"maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:46.502000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:46.501965    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"]
Nov 03 00:15:46.521672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:46.521643    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:46.571340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:46.571222    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" path="/var/lib/kubelet/pods/576d601b-8d25-454e-bbc4-e68a1d8483e4/volumes"
Nov 03 00:15:46.635334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:46.635301    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:46.846764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:46.843850    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"]
Nov 03 00:15:46.862292 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:46.862210    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde14e2d3_979f_4186_91fe_6f8630291d8a.slice/crio-9e3dfc7d7ddcf5a093c232eca4850da3bf4036fdbf5fc9efa9ca8c79f3d73308 WatchSource:0}: Error finding container 9e3dfc7d7ddcf5a093c232eca4850da3bf4036fdbf5fc9efa9ca8c79f3d73308: Status 404 returned error can't find the container with id 9e3dfc7d7ddcf5a093c232eca4850da3bf4036fdbf5fc9efa9ca8c79f3d73308
Nov 03 00:15:47.290027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.289698    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:47.358128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.358041    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n" event={"ID":"de14e2d3-979f-4186-91fe-6f8630291d8a","Type":"ContainerStarted","Data":"9e3dfc7d7ddcf5a093c232eca4850da3bf4036fdbf5fc9efa9ca8c79f3d73308"}
Nov 03 00:15:47.360269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.360226    6000 generic.go:356] "Generic (PLEG): container finished" podID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerID="96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e" exitCode=0
Nov 03 00:15:47.360386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.360285    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"
Nov 03 00:15:47.360386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.360285    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" event={"ID":"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6","Type":"ContainerDied","Data":"96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e"}
Nov 03 00:15:47.360386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.360345    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz" event={"ID":"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6","Type":"ContainerDied","Data":"7f0609531175ebbaca6fecc173e252d25b706bbb89c45c8b011a77b86a8a642b"}
Nov 03 00:15:47.360386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.360376    6000 scope.go:122] "RemoveContainer" containerID="96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e"
Nov 03 00:15:47.372592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.372566    6000 scope.go:122] "RemoveContainer" containerID="69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244"
Nov 03 00:15:47.381658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.381634    6000 scope.go:122] "RemoveContainer" containerID="96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e"
Nov 03 00:15:47.381988 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:47.381969    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e\": container with ID starting with 96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e not found: ID does not exist" containerID="96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e"
Nov 03 00:15:47.382046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.381996    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e"} err="failed to get container status \"96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e\": rpc error: code = NotFound desc = could not find container \"96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e\": container with ID starting with 96e16e1654e99024fad66a01ccdb22270f79c60a8cafb7669e1883ec7c663e5e not found: ID does not exist"
Nov 03 00:15:47.382046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.382013    6000 scope.go:122] "RemoveContainer" containerID="69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244"
Nov 03 00:15:47.382312 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:47.382293    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244\": container with ID starting with 69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244 not found: ID does not exist" containerID="69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244"
Nov 03 00:15:47.382384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.382317    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244"} err="failed to get container status \"69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244\": rpc error: code = NotFound desc = could not find container \"69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244\": container with ID starting with 69593485717a5233acd7b770847508ec237e26c0f402871782466088f582c244 not found: ID does not exist"
Nov 03 00:15:47.410585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410554    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-data\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.410730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410634    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-envoy\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.410730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410707    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-podinfo\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.410810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410742    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-token\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-token\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.410848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410824    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-kube-api-access-9dvrb\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-kube-api-access-9dvrb\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.410848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410822    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-data" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:47.410950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410851    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-socket\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.410950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410880    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istiod-ca-cert\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.410950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410920    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-credential-socket\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.411107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.410962    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-certs\") pod \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\" (UID: \"6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6\") "
Nov 03 00:15:47.411163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411135    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-socket" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411450    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istiod-ca-cert" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411458    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-credential-socket" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411535    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-certs" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411775    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411798    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411812    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411826    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.411867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.411841    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.413216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.413192    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-token" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:47.413333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.413315    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-envoy" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:15:47.413723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.413697    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-podinfo" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:15:47.413827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.413760    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-kube-api-access-9dvrb" pod "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" (UID: "6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6"). InnerVolumeSpecName "kube-api-access-9dvrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:47.512972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.512933    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.512972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.512969    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.512972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.512978    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.513467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.512987    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dvrb\" (UniqueName: \"kubernetes.io/projected/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6-kube-api-access-9dvrb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:47.704894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.704841    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"]
Nov 03 00:15:47.705529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.705497    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy"
Nov 03 00:15:47.705529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.705518    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy"
Nov 03 00:15:47.705684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.705574    6000 cpu_manager.go:414] "containerMap: removing container" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy"
Nov 03 00:15:47.705684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.705580    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="576d601b-8d25-454e-bbc4-e68a1d8483e4" containerName="istio-proxy"
Nov 03 00:15:47.705684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.705655    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy"
Nov 03 00:15:47.705684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.705669    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy"
Nov 03 00:15:47.711490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.711468    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.712337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.712312    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"]
Nov 03 00:15:47.722185 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.722159    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a96a129b-openshift-default-7d4578c85f-nwrmz"]
Nov 03 00:15:47.728422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.728398    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-8e90c5-serving-cert\""
Nov 03 00:15:47.730794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.730771    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"]
Nov 03 00:15:47.816688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.816650    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651e7bfb-6645-492e-99aa-1721bd7a6322-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.816836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.816712    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-metrics-certs\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.816836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.816799    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgz5n\" (UniqueName: \"kubernetes.io/projected/651e7bfb-6645-492e-99aa-1721bd7a6322-kube-api-access-kgz5n\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.816940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.816922    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.918373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.918340    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651e7bfb-6645-492e-99aa-1721bd7a6322-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.918601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.918397    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-metrics-certs\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.918601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.918422    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgz5n\" (UniqueName: \"kubernetes.io/projected/651e7bfb-6645-492e-99aa-1721bd7a6322-kube-api-access-kgz5n\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.918601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.918476    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.918774 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:47.918602    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-serving-cert: secret "maas-api-e2e-worker-w3-8e90c5-serving-cert" not found
Nov 03 00:15:47.918774 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:47.918678    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls podName:651e7bfb-6645-492e-99aa-1721bd7a6322 nodeName:}" failed. No retries permitted until 2026-11-03 00:15:48.418657761 +0000 UTC m=+585.974009780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls") pod "maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" (UID: "651e7bfb-6645-492e-99aa-1721bd7a6322") : secret "maas-api-e2e-worker-w3-8e90c5-serving-cert" not found
Nov 03 00:15:47.919132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.919109    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651e7bfb-6645-492e-99aa-1721bd7a6322-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.921108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.921079    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-metrics-certs\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:47.940212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:47.940187    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgz5n\" (UniqueName: \"kubernetes.io/projected/651e7bfb-6645-492e-99aa-1721bd7a6322-kube-api-access-kgz5n\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:48.366937 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.366884    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n" event={"ID":"de14e2d3-979f-4186-91fe-6f8630291d8a","Type":"ContainerStarted","Data":"acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f"}
Nov 03 00:15:48.367143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.366965    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:48.407008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.406947    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n" podStartSLOduration=2.956136521 podStartE2EDuration="3.40693337s" totalImagesPullingTime="450.796849ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:48.405418499 +0000 UTC m=+585.960770523" watchObservedRunningTime="2026-11-03 00:15:48.40693337 +0000 UTC m=+585.962285394"
Nov 03 00:15:48.428458 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.428424    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:48.431197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.431176    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls\") pod \"maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:48.566876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.566842    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" path="/var/lib/kubelet/pods/6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6/volumes"
Nov 03 00:15:48.624616 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.624534    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:48.946091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:48.946060    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"]
Nov 03 00:15:48.946950 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:48.946910    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651e7bfb_6645_492e_99aa_1721bd7a6322.slice/crio-a3778c65322023496d9b45f94392f6cbde4b5e733c928c6b5ee152e1ae3c086b WatchSource:0}: Error finding container a3778c65322023496d9b45f94392f6cbde4b5e733c928c6b5ee152e1ae3c086b: Status 404 returned error can't find the container with id a3778c65322023496d9b45f94392f6cbde4b5e733c928c6b5ee152e1ae3c086b
Nov 03 00:15:49.375641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:49.375607    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" event={"ID":"651e7bfb-6645-492e-99aa-1721bd7a6322","Type":"ContainerStarted","Data":"5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b"}
Nov 03 00:15:49.375827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:49.375669    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" event={"ID":"651e7bfb-6645-492e-99aa-1721bd7a6322","Type":"ContainerStarted","Data":"a3778c65322023496d9b45f94392f6cbde4b5e733c928c6b5ee152e1ae3c086b"}
Nov 03 00:15:49.375827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:49.375789    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:49.416849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:49.416790    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" podStartSLOduration=2.078718826 podStartE2EDuration="2.416774685s" totalImagesPullingTime="338.055859ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:49.414964284 +0000 UTC m=+586.970316307" watchObservedRunningTime="2026-11-03 00:15:49.416774685 +0000 UTC m=+586.972126708"
Nov 03 00:15:49.896237 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:49.896202    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:15:50.135321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.135281    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"]
Nov 03 00:15:50.135803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.135789    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy"
Nov 03 00:15:50.135866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.135805    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea1a000-9bcb-49d1-9e72-6ebe27cb3cd6" containerName="istio-proxy"
Nov 03 00:15:50.138063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.138044    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.153101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.153041    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-4c61c8-dockercfg-dmk6d\""
Nov 03 00:15:50.153303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.153288    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-4c61c8\""
Nov 03 00:15:50.164440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.164419    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"]
Nov 03 00:15:50.254269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.254234    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzwtp\" (UniqueName: \"kubernetes.io/projected/7c7b20a3-0524-450e-b61b-c69543af4191-kube-api-access-qzwtp\") pod \"payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.254425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.254403    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7b20a3-0524-450e-b61b-c69543af4191-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.355549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.355512    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzwtp\" (UniqueName: \"kubernetes.io/projected/7c7b20a3-0524-450e-b61b-c69543af4191-kube-api-access-qzwtp\") pod \"payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.355715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.355655    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7b20a3-0524-450e-b61b-c69543af4191-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.356657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.356628    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7b20a3-0524-450e-b61b-c69543af4191-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.379950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.379915    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzwtp\" (UniqueName: \"kubernetes.io/projected/7c7b20a3-0524-450e-b61b-c69543af4191-kube-api-access-qzwtp\") pod \"payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.449961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.449878    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:50.746910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:50.746882    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"]
Nov 03 00:15:50.748238 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:50.748213    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c7b20a3_0524_450e_b61b_c69543af4191.slice/crio-7e7f18e2f4a30c54bdf2adf5fed18f0bd96f3686f1e23fc86f4b6f82c1d1c787 WatchSource:0}: Error finding container 7e7f18e2f4a30c54bdf2adf5fed18f0bd96f3686f1e23fc86f4b6f82c1d1c787: Status 404 returned error can't find the container with id 7e7f18e2f4a30c54bdf2adf5fed18f0bd96f3686f1e23fc86f4b6f82c1d1c787
Nov 03 00:15:51.390247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.390194    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" event={"ID":"7c7b20a3-0524-450e-b61b-c69543af4191","Type":"ContainerStarted","Data":"819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e"}
Nov 03 00:15:51.390839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.390511    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:15:51.390839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.390555    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" event={"ID":"7c7b20a3-0524-450e-b61b-c69543af4191","Type":"ContainerStarted","Data":"7e7f18e2f4a30c54bdf2adf5fed18f0bd96f3686f1e23fc86f4b6f82c1d1c787"}
Nov 03 00:15:51.427676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.427630    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" podStartSLOduration=1.427615221 podStartE2EDuration="1.427615221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:51.426017555 +0000 UTC m=+588.981369578" watchObservedRunningTime="2026-11-03 00:15:51.427615221 +0000 UTC m=+588.982967245"
Nov 03 00:15:51.502601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.502561    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6" podUID="205d2a22-a46a-499c-b88e-6b658c56b368" containerName="payload-pre-processing" containerID="cri-o://588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299" gracePeriod=25
Nov 03 00:15:51.587695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.587660    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"]
Nov 03 00:15:51.590641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.590623    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.605936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.605906    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-fc9316-serving-cert\""
Nov 03 00:15:51.610765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.610738    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"]
Nov 03 00:15:51.674064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.673978    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-metrics-certs\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.674064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.674039    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-js8qq\" (UniqueName: \"kubernetes.io/projected/c7c6efed-66cc-43c5-b23c-2b09f33508a5-kube-api-access-js8qq\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.674265 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.674089    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.674265 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.674155    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c6efed-66cc-43c5-b23c-2b09f33508a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.764093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.764067    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:51.775503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.775478    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-metrics-certs\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.775652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.775515    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-js8qq\" (UniqueName: \"kubernetes.io/projected/c7c6efed-66cc-43c5-b23c-2b09f33508a5-kube-api-access-js8qq\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.775652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.775546    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.775652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.775576    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c6efed-66cc-43c5-b23c-2b09f33508a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.776675 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.776595    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c6efed-66cc-43c5-b23c-2b09f33508a5-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.778486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.778463    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-maas-api-tls\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.778728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.778706    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-metrics-certs\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.794693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.794668    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-js8qq\" (UniqueName: \"kubernetes.io/projected/c7c6efed-66cc-43c5-b23c-2b09f33508a5-kube-api-access-js8qq\") pod \"maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.876056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.876021    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/205d2a22-a46a-499c-b88e-6b658c56b368-kube-api-access-zcm9l\" (UniqueName: \"kubernetes.io/projected/205d2a22-a46a-499c-b88e-6b658c56b368-kube-api-access-zcm9l\") pod \"205d2a22-a46a-499c-b88e-6b658c56b368\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") "
Nov 03 00:15:51.876232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.876213    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/205d2a22-a46a-499c-b88e-6b658c56b368-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205d2a22-a46a-499c-b88e-6b658c56b368-plugins-config-volume\") pod \"205d2a22-a46a-499c-b88e-6b658c56b368\" (UID: \"205d2a22-a46a-499c-b88e-6b658c56b368\") "
Nov 03 00:15:51.876615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.876590    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/205d2a22-a46a-499c-b88e-6b658c56b368-plugins-config-volume" pod "205d2a22-a46a-499c-b88e-6b658c56b368" (UID: "205d2a22-a46a-499c-b88e-6b658c56b368"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:51.878312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.878289    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/205d2a22-a46a-499c-b88e-6b658c56b368-kube-api-access-zcm9l" pod "205d2a22-a46a-499c-b88e-6b658c56b368" (UID: "205d2a22-a46a-499c-b88e-6b658c56b368"). InnerVolumeSpecName "kube-api-access-zcm9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:51.903561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.903531    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:51.977240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.977213    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/205d2a22-a46a-499c-b88e-6b658c56b368-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:51.977240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:51.977243    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcm9l\" (UniqueName: \"kubernetes.io/projected/205d2a22-a46a-499c-b88e-6b658c56b368-kube-api-access-zcm9l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:52.200633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.200607    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"]
Nov 03 00:15:52.202538 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:52.202503    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7c6efed_66cc_43c5_b23c_2b09f33508a5.slice/crio-1425bcbe93e4df752524e38637426e409065cf5a25df74dd7fcb4639be0bd01a WatchSource:0}: Error finding container 1425bcbe93e4df752524e38637426e409065cf5a25df74dd7fcb4639be0bd01a: Status 404 returned error can't find the container with id 1425bcbe93e4df752524e38637426e409065cf5a25df74dd7fcb4639be0bd01a
Nov 03 00:15:52.397341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.397306    6000 generic.go:356] "Generic (PLEG): container finished" podID="205d2a22-a46a-499c-b88e-6b658c56b368" containerID="588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299" exitCode=0
Nov 03 00:15:52.397757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.397361    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"
Nov 03 00:15:52.397757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.397400    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6" event={"ID":"205d2a22-a46a-499c-b88e-6b658c56b368","Type":"ContainerDied","Data":"588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299"}
Nov 03 00:15:52.397757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.397463    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6" event={"ID":"205d2a22-a46a-499c-b88e-6b658c56b368","Type":"ContainerDied","Data":"a961a50bf98b79bb5ffd7d2d1f2bf720faa2c23dbcd5c6f8a7004f9fbe3ff992"}
Nov 03 00:15:52.397757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.397497    6000 scope.go:122] "RemoveContainer" containerID="588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299"
Nov 03 00:15:52.403107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.403081    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6" event={"ID":"c7c6efed-66cc-43c5-b23c-2b09f33508a5","Type":"ContainerStarted","Data":"1425bcbe93e4df752524e38637426e409065cf5a25df74dd7fcb4639be0bd01a"}
Nov 03 00:15:52.409240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.409225    6000 scope.go:122] "RemoveContainer" containerID="588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299"
Nov 03 00:15:52.409562 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:15:52.409541    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299\": container with ID starting with 588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299 not found: ID does not exist" containerID="588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299"
Nov 03 00:15:52.409666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.409571    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299"} err="failed to get container status \"588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299\": rpc error: code = NotFound desc = could not find container \"588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299\": container with ID starting with 588ee122224549265634a19c33bead4d3424a363620f858d49942d85a718d299 not found: ID does not exist"
Nov 03 00:15:52.444162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.444135    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"]
Nov 03 00:15:52.447568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.447544    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-e3e6ed37-7bdd4b86f5-bbvc6"]
Nov 03 00:15:52.566432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:52.566408    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="205d2a22-a46a-499c-b88e-6b658c56b368" path="/var/lib/kubelet/pods/205d2a22-a46a-499c-b88e-6b658c56b368/volumes"
Nov 03 00:15:53.196720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.196687    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"]
Nov 03 00:15:53.196976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.196922    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz" podUID="a7e21f59-6104-43d9-a6e7-862ed05c186b" containerName="maas-api" containerID="cri-o://8b995273805e3a07c0892cec319dacbb7587ccf80b7f4b9843f85c4029ca039b" gracePeriod=30
Nov 03 00:15:53.412615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.412580    6000 generic.go:356] "Generic (PLEG): container finished" podID="a7e21f59-6104-43d9-a6e7-862ed05c186b" containerID="8b995273805e3a07c0892cec319dacbb7587ccf80b7f4b9843f85c4029ca039b" exitCode=0
Nov 03 00:15:53.413010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.412643    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz" event={"ID":"a7e21f59-6104-43d9-a6e7-862ed05c186b","Type":"ContainerDied","Data":"8b995273805e3a07c0892cec319dacbb7587ccf80b7f4b9843f85c4029ca039b"}
Nov 03 00:15:53.414246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.414223    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6" event={"ID":"c7c6efed-66cc-43c5-b23c-2b09f33508a5","Type":"ContainerStarted","Data":"82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c"}
Nov 03 00:15:53.414410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.414395    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:15:53.458427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.458379    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6" podStartSLOduration=2.102154375 podStartE2EDuration="2.45836411s" totalImagesPullingTime="356.209735ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:53.455986161 +0000 UTC m=+591.011338219" watchObservedRunningTime="2026-11-03 00:15:53.45836411 +0000 UTC m=+591.013716143"
Nov 03 00:15:53.479815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.479793    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:53.598810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.598779    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-maas-api-tls\") pod \"a7e21f59-6104-43d9-a6e7-862ed05c186b\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") "
Nov 03 00:15:53.599019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.598833    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-metrics-certs\") pod \"a7e21f59-6104-43d9-a6e7-862ed05c186b\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") "
Nov 03 00:15:53.599019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.598906    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e21f59-6104-43d9-a6e7-862ed05c186b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e21f59-6104-43d9-a6e7-862ed05c186b-openshift-service-ca\") pod \"a7e21f59-6104-43d9-a6e7-862ed05c186b\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") "
Nov 03 00:15:53.599019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.598950    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e21f59-6104-43d9-a6e7-862ed05c186b-kube-api-access-sn6mz\" (UniqueName: \"kubernetes.io/projected/a7e21f59-6104-43d9-a6e7-862ed05c186b-kube-api-access-sn6mz\") pod \"a7e21f59-6104-43d9-a6e7-862ed05c186b\" (UID: \"a7e21f59-6104-43d9-a6e7-862ed05c186b\") "
Nov 03 00:15:53.599433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.599406    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e21f59-6104-43d9-a6e7-862ed05c186b-openshift-service-ca" pod "a7e21f59-6104-43d9-a6e7-862ed05c186b" (UID: "a7e21f59-6104-43d9-a6e7-862ed05c186b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:15:53.601132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.601106    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-maas-api-tls" pod "a7e21f59-6104-43d9-a6e7-862ed05c186b" (UID: "a7e21f59-6104-43d9-a6e7-862ed05c186b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:53.601887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.601863    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-metrics-certs" pod "a7e21f59-6104-43d9-a6e7-862ed05c186b" (UID: "a7e21f59-6104-43d9-a6e7-862ed05c186b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:15:53.601971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.601891    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e21f59-6104-43d9-a6e7-862ed05c186b-kube-api-access-sn6mz" pod "a7e21f59-6104-43d9-a6e7-862ed05c186b" (UID: "a7e21f59-6104-43d9-a6e7-862ed05c186b"). InnerVolumeSpecName "kube-api-access-sn6mz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:15:53.700906 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.700874    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:53.700906 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.700901    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e21f59-6104-43d9-a6e7-862ed05c186b-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:53.700906 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.700912    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e21f59-6104-43d9-a6e7-862ed05c186b-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:53.701146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.700928    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6mz\" (UniqueName: \"kubernetes.io/projected/a7e21f59-6104-43d9-a6e7-862ed05c186b-kube-api-access-sn6mz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:15:53.890390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.890357    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"]
Nov 03 00:15:53.891124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.891101    6000 cpu_manager.go:414] "containerMap: removing container" podUID="205d2a22-a46a-499c-b88e-6b658c56b368" containerName="payload-pre-processing"
Nov 03 00:15:53.891228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.891126    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="205d2a22-a46a-499c-b88e-6b658c56b368" containerName="payload-pre-processing"
Nov 03 00:15:53.891228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.891176    6000 cpu_manager.go:414] "containerMap: removing container" podUID="a7e21f59-6104-43d9-a6e7-862ed05c186b" containerName="maas-api"
Nov 03 00:15:53.891228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.891185    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e21f59-6104-43d9-a6e7-862ed05c186b" containerName="maas-api"
Nov 03 00:15:53.891562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.891542    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="205d2a22-a46a-499c-b88e-6b658c56b368" containerName="payload-pre-processing"
Nov 03 00:15:53.891652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.891576    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e21f59-6104-43d9-a6e7-862ed05c186b" containerName="maas-api"
Nov 03 00:15:53.895439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.895417    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:53.910513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.910486    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-4c61c8-serving-cert\""
Nov 03 00:15:53.953044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:53.953007    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"]
Nov 03 00:15:54.003979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.003946    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa21b75-6104-431b-ba39-a56035f517ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.004186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.003989    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5zg9\" (UniqueName: \"kubernetes.io/projected/7fa21b75-6104-431b-ba39-a56035f517ac-kube-api-access-q5zg9\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.004186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.004098    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.004366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.004324    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-metrics-certs\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.105633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.105588    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa21b75-6104-431b-ba39-a56035f517ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.105820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.105655    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5zg9\" (UniqueName: \"kubernetes.io/projected/7fa21b75-6104-431b-ba39-a56035f517ac-kube-api-access-q5zg9\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.105820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.105737    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.105949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.105884    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-metrics-certs\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.107126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.107095    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa21b75-6104-431b-ba39-a56035f517ac-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.110485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.110463    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-maas-api-tls\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.111544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.111521    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-metrics-certs\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.141946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.141867    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5zg9\" (UniqueName: \"kubernetes.io/projected/7fa21b75-6104-431b-ba39-a56035f517ac-kube-api-access-q5zg9\") pod \"maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.212461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.212431    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:54.382404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.382377    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 00:15:54.420884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.420792    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"
Nov 03 00:15:54.420884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.420799    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz" event={"ID":"a7e21f59-6104-43d9-a6e7-862ed05c186b","Type":"ContainerDied","Data":"2d31f428253f732cdbcd4a40e6d9cf9e9c6a4c0c511750b7dc543122061d0cbe"}
Nov 03 00:15:54.421437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.420947    6000 scope.go:122] "RemoveContainer" containerID="8b995273805e3a07c0892cec319dacbb7587ccf80b7f4b9843f85c4029ca039b"
Nov 03 00:15:54.471563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.471534    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"]
Nov 03 00:15:54.475298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.475272    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e3e6ed37-6966c48986-rwvwz"]
Nov 03 00:15:54.533774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.533751    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"]
Nov 03 00:15:54.535442 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:15:54.535413    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa21b75_6104_431b_ba39_a56035f517ac.slice/crio-88bf1abd02f163c8524355e8bbf954c6e87f70e811a516af250276fde98e0ca5 WatchSource:0}: Error finding container 88bf1abd02f163c8524355e8bbf954c6e87f70e811a516af250276fde98e0ca5: Status 404 returned error can't find the container with id 88bf1abd02f163c8524355e8bbf954c6e87f70e811a516af250276fde98e0ca5
Nov 03 00:15:54.566446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.566417    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e21f59-6104-43d9-a6e7-862ed05c186b" path="/var/lib/kubelet/pods/a7e21f59-6104-43d9-a6e7-862ed05c186b/volumes"
Nov 03 00:15:54.753097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:54.753014    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:55.389342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:55.389317    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 00:15:55.429818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:55.429667    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv" event={"ID":"7fa21b75-6104-431b-ba39-a56035f517ac","Type":"ContainerStarted","Data":"aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4"}
Nov 03 00:15:55.429818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:55.429775    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv" event={"ID":"7fa21b75-6104-431b-ba39-a56035f517ac","Type":"ContainerStarted","Data":"88bf1abd02f163c8524355e8bbf954c6e87f70e811a516af250276fde98e0ca5"}
Nov 03 00:15:55.430338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:55.429921    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:15:55.480868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:55.480805    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv" podStartSLOduration=2.153522752 podStartE2EDuration="2.480787038s" totalImagesPullingTime="327.264286ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:15:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:15:55.479209852 +0000 UTC m=+593.034561879" watchObservedRunningTime="2026-11-03 00:15:55.480787038 +0000 UTC m=+593.036139066"
Nov 03 00:15:59.143713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:59.143672    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:15:59.426879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:15:59.426804    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 00:16:01.442955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:01.442926    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 00:16:02.413309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:02.413279    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 00:16:06.521317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:06.521273    6000 generic.go:356] "Generic (PLEG): container finished" podID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerID="4960a94afeb68a0a51ae3656361643a1caff85e8a44e6dd6ccd2de3db27290c0" exitCode=0
Nov 03 00:16:06.521754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:06.521287    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerDied","Data":"4960a94afeb68a0a51ae3656361643a1caff85e8a44e6dd6ccd2de3db27290c0"}
Nov 03 00:16:06.521861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:06.521842    6000 scope.go:122] "RemoveContainer" containerID="4960a94afeb68a0a51ae3656361643a1caff85e8a44e6dd6ccd2de3db27290c0"
Nov 03 00:16:06.523297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:06.523230    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:06.523421 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:06.523354    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:06.523421 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:06.523408    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:07.337297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:07.337267    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:16:07.529081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:07.529035    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerStarted","Data":"4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537"}
Nov 03 00:16:08.519015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:08.518981    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:16:08.519861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:08.519834    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:08.519976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:08.519883    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:09.519444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.519406    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:09.519898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.519458    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:09.540951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.540919    6000 generic.go:356] "Generic (PLEG): container finished" podID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerID="e96de02f82f86a34da6b11d24e1416e2e1688b00ef11463548fe006b3a003032" exitCode=0
Nov 03 00:16:09.541116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.540993    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerDied","Data":"e96de02f82f86a34da6b11d24e1416e2e1688b00ef11463548fe006b3a003032"}
Nov 03 00:16:09.541430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.541417    6000 scope.go:122] "RemoveContainer" containerID="e96de02f82f86a34da6b11d24e1416e2e1688b00ef11463548fe006b3a003032"
Nov 03 00:16:09.542720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.542693    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:09.542795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.542760    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:09.542831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.542806    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:09.897245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.897211    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"]
Nov 03 00:16:09.901093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.901072    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:09.918471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.918447    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-b85343-serving-cert\""
Nov 03 00:16:09.919911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.919886    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"]
Nov 03 00:16:09.955626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.955587    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-metrics-certs\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:09.955824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.955661    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886baa1-3552-4ee4-8609-13c51431268d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:09.955824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.955745    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgzdf\" (UniqueName: \"kubernetes.io/projected/e886baa1-3552-4ee4-8609-13c51431268d-kube-api-access-xgzdf\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:09.955824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:09.955790    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.057295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.057241    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.057461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.057390    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-metrics-certs\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.057515 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.057490    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886baa1-3552-4ee4-8609-13c51431268d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.057572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.057538    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgzdf\" (UniqueName: \"kubernetes.io/projected/e886baa1-3552-4ee4-8609-13c51431268d-kube-api-access-xgzdf\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.058326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.058291    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886baa1-3552-4ee4-8609-13c51431268d-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.060154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.060134    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-maas-api-tls\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.060343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.060323    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-metrics-certs\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.076901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.076874    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgzdf\" (UniqueName: \"kubernetes.io/projected/e886baa1-3552-4ee4-8609-13c51431268d-kube-api-access-xgzdf\") pod \"maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.213975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.213893    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:10.510665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.510638    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"]
Nov 03 00:16:10.511477 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:16:10.511454    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode886baa1_3552_4ee4_8609_13c51431268d.slice/crio-c42efae3511169674abae7c1e249b2d3f6f6780c7ca7b2b126516a37b5fb7bf2 WatchSource:0}: Error finding container c42efae3511169674abae7c1e249b2d3f6f6780c7ca7b2b126516a37b5fb7bf2: Status 404 returned error can't find the container with id c42efae3511169674abae7c1e249b2d3f6f6780c7ca7b2b126516a37b5fb7bf2
Nov 03 00:16:10.519825 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.519801    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:10.520111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.519845    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:10.547855 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.547829    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerStarted","Data":"e67627834272cfb28ec16b5acf081b32cac27572eaf42b2c2bf6d772ec71b47e"}
Nov 03 00:16:10.549068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.549043    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x" event={"ID":"e886baa1-3552-4ee4-8609-13c51431268d","Type":"ContainerStarted","Data":"c42efae3511169674abae7c1e249b2d3f6f6780c7ca7b2b126516a37b5fb7bf2"}
Nov 03 00:16:10.586873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.586835    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:16:10.587045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.587022    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:10.587110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:10.587072    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:11.519853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:11.519819    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:11.520250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:11.519871    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:11.555991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:11.555960    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x" event={"ID":"e886baa1-3552-4ee4-8609-13c51431268d","Type":"ContainerStarted","Data":"7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a"}
Nov 03 00:16:11.556153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:11.556111    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:11.587377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:11.587353    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:11.587568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:11.587397    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:11.603643 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:11.603599    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x" podStartSLOduration=2.208629302 podStartE2EDuration="2.603585283s" totalImagesPullingTime="394.955981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:16:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:16:11.602450784 +0000 UTC m=+609.157802803" watchObservedRunningTime="2026-11-03 00:16:11.603585283 +0000 UTC m=+609.158937307"
Nov 03 00:16:12.519571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:12.519537    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:12.519754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:12.519605    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:12.587056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:12.587019    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:12.587606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:12.587074    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:13.330749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:13.330713    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:16:13.519391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:13.519348    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:13.519553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:13.519411    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:13.587869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:13.587778    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:13.587869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:13.587832    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:14.519653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:14.519619    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:14.519814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:14.519674    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:14.587204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:14.587171    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:14.587393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:14.587231    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:15.345305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:15.345274    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:16:15.519302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:15.519243    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:15.519486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:15.519320    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:15.587725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:15.587689    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:15.587904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:15.587752    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:15.648113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:15.648080    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:16:16.518892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:16.518859    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:16.519331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:16.518910    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:16.587455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:16.587420    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:16.587641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:16.587473    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:17.519793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:17.519757    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:17.520188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:17.519814    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:17.572267 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:17.572237    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 00:16:17.586904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:17.586877    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:17.587062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:17.586942    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:18.518897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:18.518860    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:18.519089 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:18.518963    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:18.587089 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:18.587059    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:18.587644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:18.587119    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:19.519142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:19.519110    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:19.519353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:19.519160    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:19.587764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:19.587713    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:19.588134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:19.587792    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:19.895373 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:16:19.895338    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:16:20.519833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:20.519796    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:20.520008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:20.519855    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:20.587667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:20.587635    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:20.587829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:20.587692    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:21.518869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:21.518834    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:21.519057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:21.518895    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:21.586960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:21.586919    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:21.587125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:21.586981    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:22.519871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:22.519833    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:22.520413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:22.519892    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:22.587592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:22.587558    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:22.587751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:22.587613    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:23.138769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:23.138737    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:16:23.518920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:23.518843    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:23.518920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:23.518896    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:23.587190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:23.587158    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:23.587587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:23.587211    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:24.519130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:24.519041    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:24.519130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:24.519102    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:24.587595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:24.587559    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:24.587971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:24.587610    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:25.518790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:25.518754    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:25.518979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:25.518805    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:25.587674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:25.587645    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:25.588015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:25.587693    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:26.518840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:26.518809    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:26.519069 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:26.518861    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:26.587127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:26.587096    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:26.587329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:26.587143    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:27.519190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:27.519152    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:27.519607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:27.519208    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:27.587295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:27.587230    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:27.587463 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:27.587318    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:28.518929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:28.518891    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:28.519119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:28.518952    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:28.587405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:28.587372    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:28.587784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:28.587427    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:29.519125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:29.519092    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:29.519334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:29.519151    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:29.587177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:29.587145    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:29.587366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:29.587198    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:30.519694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:30.519658    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:30.520079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:30.519712    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:30.587064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:30.587025    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:30.587231 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:30.587087    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:31.519434 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:31.519395    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:31.519677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:31.519478    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:31.587593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:31.587553    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:31.588054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:31.587617    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:31.926938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:31.926899    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:16:32.519893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:32.519859    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:32.520076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:32.519911    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:32.587542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:32.587510    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:32.587734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:32.587577    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:33.519787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:33.519747    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:33.519999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:33.519796    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:33.587653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:33.587612    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:33.587828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:33.587680    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:34.519444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:34.519409    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:34.519622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:34.519462    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:34.587657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:34.587624    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:34.587825 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:34.587675    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:35.519725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:35.519690    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:35.520109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:35.519745    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:35.587905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:35.587872    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:35.588067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:35.587925    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:36.519006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:36.518969    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:36.519203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:36.519030    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:36.587680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:36.587642    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:36.588136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:36.587702    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:37.519445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:37.519406    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:37.519637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:37.519465    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:37.519637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:37.519507    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:16:37.520021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:37.520004    6000 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537"} pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" containerMessage="Container istio-proxy failed startup probe, will be restarted"
Nov 03 00:16:37.520066 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:37.520047    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" containerID="cri-o://4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537" gracePeriod=30
Nov 03 00:16:37.587688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:37.587655    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:37.588074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:37.587703    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:38.587612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:38.587576    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:38.587799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:38.587634    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:39.587652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:39.587623    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:39.587823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:39.587671    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:39.587823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:39.587712    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:16:39.588113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:39.588078    6000 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"e67627834272cfb28ec16b5acf081b32cac27572eaf42b2c2bf6d772ec71b47e"} pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" containerMessage="Container istio-proxy failed startup probe, will be restarted"
Nov 03 00:16:39.588151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:39.588117    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" containerID="cri-o://e67627834272cfb28ec16b5acf081b32cac27572eaf42b2c2bf6d772ec71b47e" gracePeriod=30
Nov 03 00:16:42.640647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:42.640611    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:42.640960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:42.640691    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:42.640960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:42.640725    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:42.718109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:42.718074    6000 generic.go:356] "Generic (PLEG): container finished" podID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerID="4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537" exitCode=0
Nov 03 00:16:42.718243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:42.718136    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerDied","Data":"4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537"}
Nov 03 00:16:42.718243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:42.718195    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerStarted","Data":"b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790"}
Nov 03 00:16:42.718243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:42.718217    6000 scope.go:122] "RemoveContainer" containerID="4960a94afeb68a0a51ae3656361643a1caff85e8a44e6dd6ccd2de3db27290c0"
Nov 03 00:16:43.330071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:43.330046    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:16:43.519505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:43.519476    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:16:43.519674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:43.519622    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:43.519723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:43.519674    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:44.519597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.519563    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:44.520003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.519619    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:44.707240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.707209    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:44.707322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.707310    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:44.707366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.707347    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:16:44.732405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.732378    6000 generic.go:356] "Generic (PLEG): container finished" podID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerID="e67627834272cfb28ec16b5acf081b32cac27572eaf42b2c2bf6d772ec71b47e" exitCode=0
Nov 03 00:16:44.732517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.732448    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerDied","Data":"e67627834272cfb28ec16b5acf081b32cac27572eaf42b2c2bf6d772ec71b47e"}
Nov 03 00:16:44.732517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:44.732511    6000 scope.go:122] "RemoveContainer" containerID="e96de02f82f86a34da6b11d24e1416e2e1688b00ef11463548fe006b3a003032"
Nov 03 00:16:45.519247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:45.519215    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:45.519422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:45.519282    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:45.739391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:45.739353    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerStarted","Data":"8f724a75f11f39072edeca649bc0d55094f6c10488d2dbcd3b428c3094f69303"}
Nov 03 00:16:46.519598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:46.519558    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:46.519777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:46.519620    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:46.587858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:46.587826    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:16:46.588028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:46.587996    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:46.588076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:46.588039    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:47.519649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:47.519618    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:47.520049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:47.519668    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:47.587471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:47.587435    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:47.587630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:47.587492    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:48.519610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:48.519575    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:48.519785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:48.519632    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:48.587769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:48.587737    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:48.587926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:48.587795    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:49.518934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:49.518889    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:49.519147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:49.518958    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:49.587507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:49.587473    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:49.587862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:49.587523    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:49.921790 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:16:49.921755    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:16:50.520769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:50.520726    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:50.521063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:50.521030    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:50.590697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:50.590659    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:50.592024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:50.591974    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:51.519656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:51.519619    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:51.519974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:51.519942    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:51.590284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:51.587650    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:51.590284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:51.587713    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:52.526687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:52.526647    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:52.527345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:52.527314    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:52.605281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:52.601416    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:52.605281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:52.601477    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:53.519655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:53.519615    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:53.520059 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:53.520028    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:53.591659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:53.591620    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:53.592241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:53.592211    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:54.522282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:54.521683    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:54.522282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:54.521764    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:54.590642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:54.590604    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 03 00:16:54.590934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:54.590907    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 03 00:16:55.519875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:55.519832    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 03 00:16:55.520465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:55.519897    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 03 00:16:56.129162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:56.129134    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:16:56.129482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:56.129458    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:16:56.130309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:56.130289    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 00:16:56.523448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:56.523383    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:16:56.524530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:56.524507    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:16:59.525818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:16:59.525782    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:17:04.876235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.876198    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"]
Nov 03 00:17:04.881901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.881872    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:04.898531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.898505    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"]
Nov 03 00:17:04.907733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.907689    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"openshift-service-ca.crt\""
Nov 03 00:17:04.907733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.907701    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"kube-root-ca.crt\""
Nov 03 00:17:04.907919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.907743    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"premium-w0-fc9316-kserve-self-signed-certs\""
Nov 03 00:17:04.907919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.907771    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"default-dockercfg-qffjg\""
Nov 03 00:17:04.952776 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.952741    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tmp-dir\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:04.952966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.952854    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-model-cache\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:04.952966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.952903    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-home\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:04.952966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.952930    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvrth\" (UniqueName: \"kubernetes.io/projected/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-kube-api-access-rvrth\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:04.953142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.953063    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-dshm\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:04.953142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:04.953109    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tls-certs\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.056957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.056896    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-dshm\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.056957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.056957    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tls-certs\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.057239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.057052    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tmp-dir\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.057239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.057137    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-model-cache\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.057239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.057175    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-home\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.057239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.057205    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvrth\" (UniqueName: \"kubernetes.io/projected/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-kube-api-access-rvrth\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.057819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.057786    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tmp-dir\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.057910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.057849    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-model-cache\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.058095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.058075    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-home\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.061074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.061044    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-dshm\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.066449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.066387    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tls-certs\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.078886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.078853    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvrth\" (UniqueName: \"kubernetes.io/projected/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-kube-api-access-rvrth\") pod \"premium-w0-fc9316-kserve-694779777d-9qlr6\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") " pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.195161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.195078    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:05.492528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.492501    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"]
Nov 03 00:17:05.494633 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:17:05.494604    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf877ab10_7fe1_4309_95d4_769dc5b0ae0e.slice/crio-79cbf0bdc64bb04e23b572cdc2d3d1ba3d33841950c67bf0ce2ce2c41ce9e558 WatchSource:0}: Error finding container 79cbf0bdc64bb04e23b572cdc2d3d1ba3d33841950c67bf0ce2ce2c41ce9e558: Status 404 returned error can't find the container with id 79cbf0bdc64bb04e23b572cdc2d3d1ba3d33841950c67bf0ce2ce2c41ce9e558
Nov 03 00:17:05.916327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.916288    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" event={"ID":"f877ab10-7fe1-4309-95d4-769dc5b0ae0e","Type":"ContainerStarted","Data":"6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca"}
Nov 03 00:17:05.916732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.916342    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" event={"ID":"f877ab10-7fe1-4309-95d4-769dc5b0ae0e","Type":"ContainerStarted","Data":"79cbf0bdc64bb04e23b572cdc2d3d1ba3d33841950c67bf0ce2ce2c41ce9e558"}
Nov 03 00:17:05.954103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:05.954034    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podStartSLOduration=1.954000661 podStartE2EDuration="1.954000661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:05.952702274 +0000 UTC m=+663.508054299" watchObservedRunningTime="2026-11-03 00:17:05.954000661 +0000 UTC m=+663.509352685"
Nov 03 00:17:06.196180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:06.196089    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:08.067346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:08.067312    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:17:11.393133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.393099    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"]
Nov 03 00:17:11.397572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.397549    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.412454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.412426    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"openshift-service-ca.crt\""
Nov 03 00:17:11.423977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.423948    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"kube-root-ca.crt\""
Nov 03 00:17:11.424130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.424052    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"premium-w2-4c61c8-kserve-self-signed-certs\""
Nov 03 00:17:11.424130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.424092    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-4c61c8\"/\"default-dockercfg-lqdnc\""
Nov 03 00:17:11.425736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.425717    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"]
Nov 03 00:17:11.547507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.547475    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-home\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.547507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.547510    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-tmp-dir\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.547724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.547658    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-dshm\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.547724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.547701    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02281a18-5373-45a1-89a6-25f6ca6e2911-tls-certs\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.547799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.547745    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-model-cache\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.547832 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.547818    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvl9l\" (UniqueName: \"kubernetes.io/projected/02281a18-5373-45a1-89a6-25f6ca6e2911-kube-api-access-tvl9l\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.648818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.648737    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-home\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.648818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.648773    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-tmp-dir\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.649027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.648823    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-dshm\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.649027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.648847    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02281a18-5373-45a1-89a6-25f6ca6e2911-tls-certs\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.649027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.648881    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-model-cache\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.649027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.648910    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvl9l\" (UniqueName: \"kubernetes.io/projected/02281a18-5373-45a1-89a6-25f6ca6e2911-kube-api-access-tvl9l\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.649235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.649188    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-home\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.649342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.649228    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-tmp-dir\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.649400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.649342    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-model-cache\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.651431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.651405    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-dshm\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.651557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.651548    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02281a18-5373-45a1-89a6-25f6ca6e2911-tls-certs\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.668502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.668472    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvl9l\" (UniqueName: \"kubernetes.io/projected/02281a18-5373-45a1-89a6-25f6ca6e2911-kube-api-access-tvl9l\") pod \"premium-w2-4c61c8-kserve-695cf5d76f-xhtv6\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") " pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:11.711883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:11.711846    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:12.013156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:12.013125    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"]
Nov 03 00:17:12.014544 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:17:12.014512    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02281a18_5373_45a1_89a6_25f6ca6e2911.slice/crio-18c3709e82755bfb807da7cf0d4a43dfacda5acb3f1f4e8e7bd2837baa2a358a WatchSource:0}: Error finding container 18c3709e82755bfb807da7cf0d4a43dfacda5acb3f1f4e8e7bd2837baa2a358a: Status 404 returned error can't find the container with id 18c3709e82755bfb807da7cf0d4a43dfacda5acb3f1f4e8e7bd2837baa2a358a
Nov 03 00:17:12.956880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:12.956847    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" event={"ID":"02281a18-5373-45a1-89a6-25f6ca6e2911","Type":"ContainerStarted","Data":"cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268"}
Nov 03 00:17:12.957314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:12.956891    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" event={"ID":"02281a18-5373-45a1-89a6-25f6ca6e2911","Type":"ContainerStarted","Data":"18c3709e82755bfb807da7cf0d4a43dfacda5acb3f1f4e8e7bd2837baa2a358a"}
Nov 03 00:17:12.997361 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:12.997312    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podStartSLOduration=1.99729676 podStartE2EDuration="1.99729676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:12.995597185 +0000 UTC m=+670.550949210" watchObservedRunningTime="2026-11-03 00:17:12.99729676 +0000 UTC m=+670.552648784"
Nov 03 00:17:13.712659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:13.712631    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:15.195312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:15.195280    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:15.208329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:15.208308    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:15.983498 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:15.983471    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:17:19.895095 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:17:19.895051    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:17:21.712551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:21.712515    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:21.726366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:21.726339    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:22.024037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:22.023953    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:17:22.927368 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:22.927335    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:17:32.806590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.806554    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"]
Nov 03 00:17:32.811083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.811061    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.825525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.825505    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-99bdb8-openshift-default-dockercfg-dq2f2\""
Nov 03 00:17:32.839337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.839308    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"]
Nov 03 00:17:32.885519 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885483    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.885718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885571    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c127e-840d-4daa-afbb-c266e2cc5156-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.885718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885610    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.885829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885720    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.885829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885772    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.885929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885861    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.885929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885885    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.886019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.885994    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.886072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.886037    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrsx\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-kube-api-access-4lrsx\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.986639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.986610    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.986826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.986652    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrsx\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-kube-api-access-4lrsx\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.986826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.986712    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.986826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.986732    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c127e-840d-4daa-afbb-c266e2cc5156-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.986826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.986763    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.986840    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.986970    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.987043    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987207 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.987124    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987207 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.987160    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.987375    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.987473    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.987527    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c127e-840d-4daa-afbb-c266e2cc5156-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.987650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.987634    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.989239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.989214    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:32.989734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:32.989715    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:33.005718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:33.005695    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:33.006140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:33.006115    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrsx\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-kube-api-access-4lrsx\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:33.125968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:33.125937    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:33.438759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:33.438730    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"]
Nov 03 00:17:33.439547 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:17:33.439517    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae8c127e_840d_4daa_afbb_c266e2cc5156.slice/crio-89a9a68c59a16699e68e45b5fcb79f64ee7344198e9d31ac5a9c43fa7c8ec527 WatchSource:0}: Error finding container 89a9a68c59a16699e68e45b5fcb79f64ee7344198e9d31ac5a9c43fa7c8ec527: Status 404 returned error can't find the container with id 89a9a68c59a16699e68e45b5fcb79f64ee7344198e9d31ac5a9c43fa7c8ec527
Nov 03 00:17:33.441954 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:33.441922    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:17:33.442021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:33.441990    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:17:33.442057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:33.442018    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:17:34.077269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:34.077218    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" event={"ID":"ae8c127e-840d-4daa-afbb-c266e2cc5156","Type":"ContainerStarted","Data":"69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd"}
Nov 03 00:17:34.077734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:34.077318    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" event={"ID":"ae8c127e-840d-4daa-afbb-c266e2cc5156","Type":"ContainerStarted","Data":"89a9a68c59a16699e68e45b5fcb79f64ee7344198e9d31ac5a9c43fa7c8ec527"}
Nov 03 00:17:34.126605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:34.126565    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:34.128243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:34.128218    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:17:34.128349 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:34.128300    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 00:17:34.146155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:34.146107    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podStartSLOduration=2.146093637 podStartE2EDuration="2.146093637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:17:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:17:34.132909747 +0000 UTC m=+691.688261770" watchObservedRunningTime="2026-11-03 00:17:34.146093637 +0000 UTC m=+691.701445661"
Nov 03 00:17:35.127159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:35.127125    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:17:35.127601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:35.127176    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 00:17:36.127103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:36.127071    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:17:36.127372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:36.127123    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 00:17:36.858655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:36.858620    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:17:37.127034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:37.126948    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:17:37.127034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:37.126999    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 00:17:38.127199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:38.127152    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:17:38.127762 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:38.127221    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 00:17:39.126830 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:39.126797    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body=
Nov 03 00:17:39.127027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:39.126854    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused"
Nov 03 00:17:40.806278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:40.806208    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:40.807687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:40.807665    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:40.809884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:40.809860    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:17:49.895165 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:17:49.895118    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:17:54.176999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:17:54.176956    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:18:17.797785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.797697    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"]
Nov 03 00:18:17.801709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.801683    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.817620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.817597    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b85343-openshift-default-dockercfg-jnjww\""
Nov 03 00:18:17.831386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.831355    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"]
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837456    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2b1336-e825-47bd-b58a-a72c71d446fb-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837515    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837556    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdb6m\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-kube-api-access-fdb6m\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837582    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837627    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837706    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837747    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837779    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.839286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.837814    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.938764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.938724    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2b1336-e825-47bd-b58a-a72c71d446fb-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.938981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.938786    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.938981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.938825    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdb6m\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-kube-api-access-fdb6m\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.938981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.938851    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.938981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.938896    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939031    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939081    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939112    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939148    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939330    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939566    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939597    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939703 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939644    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2b1336-e825-47bd-b58a-a72c71d446fb-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.939703 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.939657    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.941505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.941488    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.941719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.941701    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.959443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.959402    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:17.959600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:17.959501    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdb6m\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-kube-api-access-fdb6m\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:18.115654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:18.115618    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:18.484232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:18.484200    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"]
Nov 03 00:18:18.485668 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:18:18.485643    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff2b1336_e825_47bd_b58a_a72c71d446fb.slice/crio-381ab81a2534d6255df63d5936e139b65ddab3dca6e4f3966dbd6efd4d490e33 WatchSource:0}: Error finding container 381ab81a2534d6255df63d5936e139b65ddab3dca6e4f3966dbd6efd4d490e33: Status 404 returned error can't find the container with id 381ab81a2534d6255df63d5936e139b65ddab3dca6e4f3966dbd6efd4d490e33
Nov 03 00:18:18.489047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:18.489003    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:18:18.489146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:18.489088    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:18:18.489146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:18.489135    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:18:19.351682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:19.351648    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" event={"ID":"ff2b1336-e825-47bd-b58a-a72c71d446fb","Type":"ContainerStarted","Data":"25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333"}
Nov 03 00:18:19.352097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:19.351729    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" event={"ID":"ff2b1336-e825-47bd-b58a-a72c71d446fb","Type":"ContainerStarted","Data":"381ab81a2534d6255df63d5936e139b65ddab3dca6e4f3966dbd6efd4d490e33"}
Nov 03 00:18:19.484485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:19.484421    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podStartSLOduration=2.484403967 podStartE2EDuration="2.484403967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:19.4842063 +0000 UTC m=+737.039558346" watchObservedRunningTime="2026-11-03 00:18:19.484403967 +0000 UTC m=+737.039755991"
Nov 03 00:18:19.921582 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:18:19.921545    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:18:20.117283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:20.116731    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:20.118437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:20.118409    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:18:20.118562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:20.118470    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 00:18:21.120280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:21.119643    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:18:21.120280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:21.119709    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 00:18:22.116953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:22.116918    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:18:22.117231 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:22.117210    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 00:18:23.117089 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:23.117049    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:18:23.117596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:23.117131    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 00:18:24.116783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:24.116747    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 03 00:18:24.116990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:24.116807    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 03 00:18:24.967641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:24.967595    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:18:25.281799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:25.281706    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:25.282948 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:25.282922    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:25.284309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:25.284290    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:18:42.934677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:42.934650    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:18:47.908896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.908865    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"]
Nov 03 00:18:47.912615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.912593    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.944022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.943982    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"]
Nov 03 00:18:47.990876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.990834    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.990886    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.990988    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.991022    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.991122    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.991152    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvvh\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-kube-api-access-dbvvh\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.991190    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.991330    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:47.991402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:47.991362    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092310    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092349    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092458    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092486    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092511    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092662    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092760    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092816    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvvh\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-kube-api-access-dbvvh\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092892    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.092983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.092911    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.093365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.093320    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.093490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.093368    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.093561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.093505    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.093610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.093551    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.094882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.094865    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.095130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.095114    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.110834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.110812    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.111094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.111074    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvvh\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-kube-api-access-dbvvh\") pod \"maas-default-gateway-openshift-default-6cc458d887-n4k2n\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.230299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.230203    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:48.548589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.548551    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"]
Nov 03 00:18:48.549101 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:18:48.549077    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3101c60a_bbf0_41d3_99cf_4f81bff4bff8.slice/crio-e10c728252853d7e54d59425248d24aab2a9fc2c7aeed3d0f4c4109dd11dbe85 WatchSource:0}: Error finding container e10c728252853d7e54d59425248d24aab2a9fc2c7aeed3d0f4c4109dd11dbe85: Status 404 returned error can't find the container with id e10c728252853d7e54d59425248d24aab2a9fc2c7aeed3d0f4c4109dd11dbe85
Nov 03 00:18:48.551616 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.551587    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:18:48.551692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.551653    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:18:48.551692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:48.551680    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:18:49.553920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:49.549021    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" event={"ID":"3101c60a-bbf0-41d3-99cf-4f81bff4bff8","Type":"ContainerStarted","Data":"16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180"}
Nov 03 00:18:49.553920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:49.549088    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" event={"ID":"3101c60a-bbf0-41d3-99cf-4f81bff4bff8","Type":"ContainerStarted","Data":"e10c728252853d7e54d59425248d24aab2a9fc2c7aeed3d0f4c4109dd11dbe85"}
Nov 03 00:18:49.620916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:49.620861    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podStartSLOduration=2.620847133 podStartE2EDuration="2.620847133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:49.608729538 +0000 UTC m=+767.164081563" watchObservedRunningTime="2026-11-03 00:18:49.620847133 +0000 UTC m=+767.176199160"
Nov 03 00:18:49.918110 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:18:49.918062    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:18:50.230663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:50.230584    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:50.231989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:50.231961    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n4k2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:18:50.232112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:50.232013    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 00:18:51.230833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:51.230796    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n4k2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:18:51.231367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:51.230860    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 00:18:52.230916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:52.230877    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n4k2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:18:52.231398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:52.230939    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 00:18:53.240568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:53.235045    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n4k2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:18:53.240568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:53.235145    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 00:18:54.230806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.230767    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n4k2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:18:54.231120 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.230822    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 00:18:54.692586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.692542    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"]
Nov 03 00:18:54.697459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.697435    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.716369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.716335    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"]
Nov 03 00:18:54.724599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.724572    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-fc9316\"/\"distinct-w0-fc9316-kserve-self-signed-certs\""
Nov 03 00:18:54.778760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.778731    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65fc04b9-f587-4c16-af73-efb2521cec63-tls-certs\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.778929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.778783    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-tmp-dir\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.779009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.778986    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-home\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.779062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.779050    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-dshm\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.779164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.779151    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhc9\" (UniqueName: \"kubernetes.io/projected/65fc04b9-f587-4c16-af73-efb2521cec63-kube-api-access-khhc9\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.779228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.779189    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-model-cache\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.883465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.883424    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-model-cache\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.883752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.883727    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65fc04b9-f587-4c16-af73-efb2521cec63-tls-certs\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.883925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.883908    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-tmp-dir\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.884124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.884107    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-home\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.884324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.884305    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-dshm\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.884528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.884509    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhc9\" (UniqueName: \"kubernetes.io/projected/65fc04b9-f587-4c16-af73-efb2521cec63-kube-api-access-khhc9\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.889616 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.885401    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-model-cache\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.889767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.887595    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-tmp-dir\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.889882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.888217    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-home\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.900288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.898498    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-dshm\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.913639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.913607    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhc9\" (UniqueName: \"kubernetes.io/projected/65fc04b9-f587-4c16-af73-efb2521cec63-kube-api-access-khhc9\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:54.914186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:54.914159    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65fc04b9-f587-4c16-af73-efb2521cec63-tls-certs\") pod \"distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") " pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:55.011933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:55.011842    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:55.231452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:55.231416    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n4k2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:18:55.231635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:55.231468    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 00:18:55.354578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:55.354551    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"]
Nov 03 00:18:55.356685 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:18:55.356650    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod65fc04b9_f587_4c16_af73_efb2521cec63.slice/crio-f8aa96241fcba8c1a32269bd8c06aa30a8927bf8a6e345265fd4c092db989f15 WatchSource:0}: Error finding container f8aa96241fcba8c1a32269bd8c06aa30a8927bf8a6e345265fd4c092db989f15: Status 404 returned error can't find the container with id f8aa96241fcba8c1a32269bd8c06aa30a8927bf8a6e345265fd4c092db989f15
Nov 03 00:18:55.585713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:55.585607    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" event={"ID":"65fc04b9-f587-4c16-af73-efb2521cec63","Type":"ContainerStarted","Data":"5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa"}
Nov 03 00:18:55.585713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:55.585653    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" event={"ID":"65fc04b9-f587-4c16-af73-efb2521cec63","Type":"ContainerStarted","Data":"f8aa96241fcba8c1a32269bd8c06aa30a8927bf8a6e345265fd4c092db989f15"}
Nov 03 00:18:55.623608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:55.623529    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podStartSLOduration=1.623510525 podStartE2EDuration="1.623510525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:18:55.622776117 +0000 UTC m=+773.178128140" watchObservedRunningTime="2026-11-03 00:18:55.623510525 +0000 UTC m=+773.178862548"
Nov 03 00:18:56.012206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:56.012169    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:18:56.231521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:56.231480    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-n4k2n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 03 00:18:56.231687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:56.231554    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 03 00:18:57.246006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:57.245972    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:57.246537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:57.246350    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:18:57.247307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:18:57.247250    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:19:02.160466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:02.160436    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:19:03.035287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.035221    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"]
Nov 03 00:19:03.039452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.039428    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.056285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.056239    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8e90c5-openshift-default-dockercfg-6mbt4\""
Nov 03 00:19:03.076711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.076675    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"]
Nov 03 00:19:03.096778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.096742    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17540b0-b06d-48f4-bc53-628863213789-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.097130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.097105    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.097309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.097291    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17540b0-b06d-48f4-bc53-628863213789-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.097469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.097450    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.097635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.097618    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.097759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.097743    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsqxs\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-kube-api-access-xsqxs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.097894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.097878    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.098012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.097998    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.098147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.098132    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.199963    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200052    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200085    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsqxs\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-kube-api-access-xsqxs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200123    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200146    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200189    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200234    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17540b0-b06d-48f4-bc53-628863213789-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200385    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.200620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.200417    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17540b0-b06d-48f4-bc53-628863213789-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.201334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.201301    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.201688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.201669    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.201809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.201705    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.202025 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.201993    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17540b0-b06d-48f4-bc53-628863213789-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.202091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.202002    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.214102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.206085    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.214102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.206528    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17540b0-b06d-48f4-bc53-628863213789-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.222427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.222358    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.222579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.222428    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsqxs\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-kube-api-access-xsqxs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.356968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.356934    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:03.715362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.714905    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"]
Nov 03 00:19:03.748556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.747951    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:19:03.748556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.748046    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:19:03.748556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:03.748089    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:19:04.662291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:04.662235    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" event={"ID":"d17540b0-b06d-48f4-bc53-628863213789","Type":"ContainerStarted","Data":"816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400"}
Nov 03 00:19:04.662812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:04.662342    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" event={"ID":"d17540b0-b06d-48f4-bc53-628863213789","Type":"ContainerStarted","Data":"ecc800562d7ae82bd05e56277e9dc04fc46f860f79f20b9c7990d9b85c065b64"}
Nov 03 00:19:04.736349 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:04.736294    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podStartSLOduration=2.736276685 podStartE2EDuration="2.736276685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:04.722775545 +0000 UTC m=+782.278127569" watchObservedRunningTime="2026-11-03 00:19:04.736276685 +0000 UTC m=+782.291628708"
Nov 03 00:19:05.012817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.012731    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:19:05.026417 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.026388    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:19:05.358074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.358046    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:05.359759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.359731    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:19:05.359826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.359791    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 00:19:05.679228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.679148    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:19:05.842340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.842302    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"]
Nov 03 00:19:05.847074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.847052    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:05.861816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.861792    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"kube-root-ca.crt\""
Nov 03 00:19:05.862597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.862555    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"openshift-service-ca.crt\""
Nov 03 00:19:05.863189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.863170    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"default-dockercfg-79f5r\""
Nov 03 00:19:05.863351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.863280    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"e2e-conflict-b-5f6115a4-kserve-self-signed-certs\""
Nov 03 00:19:05.864499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.864448    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"]
Nov 03 00:19:05.958286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.958173    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f965s\" (UniqueName: \"kubernetes.io/projected/5bfeffdf-2d61-4615-9b38-d299d61f7209-kube-api-access-f965s\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:05.958286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.958236    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-dshm\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:05.958558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.958305    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-tmp-dir\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:05.958558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.958430    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-home\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:05.958558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.958482    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-model-cache\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:05.958558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:05.958512    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfeffdf-2d61-4615-9b38-d299d61f7209-tls-certs\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.062690    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-home\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.062769    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-model-cache\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.062809    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfeffdf-2d61-4615-9b38-d299d61f7209-tls-certs\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.062934    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f965s\" (UniqueName: \"kubernetes.io/projected/5bfeffdf-2d61-4615-9b38-d299d61f7209-kube-api-access-f965s\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.063000    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-dshm\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.063025    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-tmp-dir\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.063605    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-tmp-dir\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.064105    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-home\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.064941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.064320    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-model-cache\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.072468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.067889    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-dshm\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.072468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.068023    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfeffdf-2d61-4615-9b38-d299d61f7209-tls-certs\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.094325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.094292    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f965s\" (UniqueName: \"kubernetes.io/projected/5bfeffdf-2d61-4615-9b38-d299d61f7209-kube-api-access-f965s\") pod \"e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") " pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.161181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.161138    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:06.357943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.357909    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:19:06.358139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.357977    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 00:19:06.485910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.485858    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"]
Nov 03 00:19:06.675232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.675199    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8" event={"ID":"5bfeffdf-2d61-4615-9b38-d299d61f7209","Type":"ContainerStarted","Data":"993c6193fff99e3c3789989388b35856217d1c7cc098c99752234a7690dd8d94"}
Nov 03 00:19:06.675232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.675238    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8" event={"ID":"5bfeffdf-2d61-4615-9b38-d299d61f7209","Type":"ContainerStarted","Data":"1c74c5ef59af244a674b86a4f94f12702f48e2ec3ab913c38951282f025aa121"}
Nov 03 00:19:06.729684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:06.729620    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8" podStartSLOduration=1.7296023470000002 podStartE2EDuration="1.729602347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:19:06.729341301 +0000 UTC m=+784.284693317" watchObservedRunningTime="2026-11-03 00:19:06.729602347 +0000 UTC m=+784.284954374"
Nov 03 00:19:07.161890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:07.161853    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:07.358352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:07.358313    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:19:07.358502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:07.358381    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 00:19:08.358537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:08.358501    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:19:08.358901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:08.358552    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 00:19:09.357915 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:09.357878    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:19:09.358109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:09.357952    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 00:19:10.357568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:10.357531    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 03 00:19:10.358004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:10.357597    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 03 00:19:11.362731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:11.362700    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:11.363218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:11.363069    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:11.363891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:11.363872    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 00:19:16.162143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:16.162110    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:16.174799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:16.174780    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:16.787355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:16.787326    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:19.942989 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:19:19.942889    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:19:24.643901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:24.643871    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:19:38.761974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:38.761940    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"]
Nov 03 00:19:45.944652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:45.944568    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:19:49.960764 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:19:49.960695    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:19:53.784770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:53.784714    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8" podUID="5bfeffdf-2d61-4615-9b38-d299d61f7209" containerName="main" containerID="cri-o://993c6193fff99e3c3789989388b35856217d1c7cc098c99752234a7690dd8d94" gracePeriod=45
Nov 03 00:19:53.789091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:53.789059    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8" podUID="5bfeffdf-2d61-4615-9b38-d299d61f7209" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": read tcp 10.134.0.2:56278->10.134.0.65:8000: read: connection reset by peer"
Nov 03 00:19:54.035730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.035666    6000 generic.go:356] "Generic (PLEG): container finished" podID="5bfeffdf-2d61-4615-9b38-d299d61f7209" containerID="993c6193fff99e3c3789989388b35856217d1c7cc098c99752234a7690dd8d94" exitCode=0
Nov 03 00:19:54.035730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.035688    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8" event={"ID":"5bfeffdf-2d61-4615-9b38-d299d61f7209","Type":"ContainerDied","Data":"993c6193fff99e3c3789989388b35856217d1c7cc098c99752234a7690dd8d94"}
Nov 03 00:19:54.035892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.035731    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8" event={"ID":"5bfeffdf-2d61-4615-9b38-d299d61f7209","Type":"ContainerDied","Data":"1c74c5ef59af244a674b86a4f94f12702f48e2ec3ab913c38951282f025aa121"}
Nov 03 00:19:54.035892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.035743    6000 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c74c5ef59af244a674b86a4f94f12702f48e2ec3ab913c38951282f025aa121"
Nov 03 00:19:54.048728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.048705    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:54.202851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.202813    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-dshm\") pod \"5bfeffdf-2d61-4615-9b38-d299d61f7209\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") "
Nov 03 00:19:54.203052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.202859    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-home\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-home\") pod \"5bfeffdf-2d61-4615-9b38-d299d61f7209\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") "
Nov 03 00:19:54.203052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.202891    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-model-cache\") pod \"5bfeffdf-2d61-4615-9b38-d299d61f7209\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") "
Nov 03 00:19:54.203052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.202940    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bfeffdf-2d61-4615-9b38-d299d61f7209-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfeffdf-2d61-4615-9b38-d299d61f7209-tls-certs\") pod \"5bfeffdf-2d61-4615-9b38-d299d61f7209\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") "
Nov 03 00:19:54.203052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.202969    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-tmp-dir\") pod \"5bfeffdf-2d61-4615-9b38-d299d61f7209\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") "
Nov 03 00:19:54.203318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.203126    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bfeffdf-2d61-4615-9b38-d299d61f7209-kube-api-access-f965s\" (UniqueName: \"kubernetes.io/projected/5bfeffdf-2d61-4615-9b38-d299d61f7209-kube-api-access-f965s\") pod \"5bfeffdf-2d61-4615-9b38-d299d61f7209\" (UID: \"5bfeffdf-2d61-4615-9b38-d299d61f7209\") "
Nov 03 00:19:54.203318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.203194    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-model-cache" pod "5bfeffdf-2d61-4615-9b38-d299d61f7209" (UID: "5bfeffdf-2d61-4615-9b38-d299d61f7209"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:54.203318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.203209    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-home" pod "5bfeffdf-2d61-4615-9b38-d299d61f7209" (UID: "5bfeffdf-2d61-4615-9b38-d299d61f7209"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:54.203318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.203303    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-tmp-dir" pod "5bfeffdf-2d61-4615-9b38-d299d61f7209" (UID: "5bfeffdf-2d61-4615-9b38-d299d61f7209"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:54.203762 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.203633    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:19:54.203875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.203847    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:19:54.203921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.203880    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:19:54.205298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.205277    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-dshm" pod "5bfeffdf-2d61-4615-9b38-d299d61f7209" (UID: "5bfeffdf-2d61-4615-9b38-d299d61f7209"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:19:54.205445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.205424    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bfeffdf-2d61-4615-9b38-d299d61f7209-tls-certs" pod "5bfeffdf-2d61-4615-9b38-d299d61f7209" (UID: "5bfeffdf-2d61-4615-9b38-d299d61f7209"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:19:54.205516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.205475    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bfeffdf-2d61-4615-9b38-d299d61f7209-kube-api-access-f965s" pod "5bfeffdf-2d61-4615-9b38-d299d61f7209" (UID: "5bfeffdf-2d61-4615-9b38-d299d61f7209"). InnerVolumeSpecName "kube-api-access-f965s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:19:54.305082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.305007    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bfeffdf-2d61-4615-9b38-d299d61f7209-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:19:54.305082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.305032    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bfeffdf-2d61-4615-9b38-d299d61f7209-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:19:54.305082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:54.305043    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f965s\" (UniqueName: \"kubernetes.io/projected/5bfeffdf-2d61-4615-9b38-d299d61f7209-kube-api-access-f965s\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:19:55.041164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:55.041134    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"
Nov 03 00:19:55.088836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:55.088791    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"]
Nov 03 00:19:55.100162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:55.100134    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/e2e-conflict-b-5f6115a4-kserve-7fb776794b-c7bz8"]
Nov 03 00:19:56.566925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:19:56.566883    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bfeffdf-2d61-4615-9b38-d299d61f7209" path="/var/lib/kubelet/pods/5bfeffdf-2d61-4615-9b38-d299d61f7209/volumes"
Nov 03 00:20:01.222557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.222518    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"]
Nov 03 00:20:01.223303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.223282    6000 cpu_manager.go:414] "containerMap: removing container" podUID="5bfeffdf-2d61-4615-9b38-d299d61f7209" containerName="main"
Nov 03 00:20:01.223405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.223305    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bfeffdf-2d61-4615-9b38-d299d61f7209" containerName="main"
Nov 03 00:20:01.223455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.223424    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bfeffdf-2d61-4615-9b38-d299d61f7209" containerName="main"
Nov 03 00:20:01.228107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.228089    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.253398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.253371    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"default-dockercfg-79f5r\""
Nov 03 00:20:01.253398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.253374    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"openshift-service-ca.crt\""
Nov 03 00:20:01.253578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.253400    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"kube-root-ca.crt\""
Nov 03 00:20:01.253578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.253419    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8e90c5\"/\"distinct-w3-8e90c5-kserve-self-signed-certs\""
Nov 03 00:20:01.255492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.255472    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"]
Nov 03 00:20:01.304609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.304580    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-model-cache\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.304802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.304614    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxlbw\" (UniqueName: \"kubernetes.io/projected/141ac223-111f-450f-b637-126126d23bd5-kube-api-access-cxlbw\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.304802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.304651    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-tmp-dir\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.304905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.304816    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141ac223-111f-450f-b637-126126d23bd5-tls-certs\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.304946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.304926    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-dshm\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.305020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.305004    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-home\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.407082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.407037    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-home\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.407432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.407409    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-model-cache\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.416416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.407832    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-home\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.416657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.416631    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxlbw\" (UniqueName: \"kubernetes.io/projected/141ac223-111f-450f-b637-126126d23bd5-kube-api-access-cxlbw\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.416804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.416783    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-tmp-dir\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.417020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.417004    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141ac223-111f-450f-b637-126126d23bd5-tls-certs\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.450103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.407881    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-model-cache\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.450103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.417653    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-tmp-dir\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.450893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.450838    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-dshm\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.474500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.474080    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxlbw\" (UniqueName: \"kubernetes.io/projected/141ac223-111f-450f-b637-126126d23bd5-kube-api-access-cxlbw\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.475179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.475148    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141ac223-111f-450f-b637-126126d23bd5-tls-certs\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.475715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.475693    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-dshm\") pod \"distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") " pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.541880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.541843    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:01.856102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:01.856074    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"]
Nov 03 00:20:01.856958 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:20:01.856937    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod141ac223_111f_450f_b637_126126d23bd5.slice/crio-f57792832a7db01b19131dae3fd968ad3cf5a824856279a204b7b38af2b2f09d WatchSource:0}: Error finding container f57792832a7db01b19131dae3fd968ad3cf5a824856279a204b7b38af2b2f09d: Status 404 returned error can't find the container with id f57792832a7db01b19131dae3fd968ad3cf5a824856279a204b7b38af2b2f09d
Nov 03 00:20:02.081003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:02.080967    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" event={"ID":"141ac223-111f-450f-b637-126126d23bd5","Type":"ContainerStarted","Data":"af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1"}
Nov 03 00:20:02.081003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:02.081012    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" event={"ID":"141ac223-111f-450f-b637-126126d23bd5","Type":"ContainerStarted","Data":"f57792832a7db01b19131dae3fd968ad3cf5a824856279a204b7b38af2b2f09d"}
Nov 03 00:20:02.119608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:02.119505    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podStartSLOduration=1.119488849 podStartE2EDuration="1.119488849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:02.117346471 +0000 UTC m=+839.672698496" watchObservedRunningTime="2026-11-03 00:20:02.119488849 +0000 UTC m=+839.674840879"
Nov 03 00:20:02.542022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:02.541949    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:03.085405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.085374    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"]
Nov 03 00:20:03.089831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.089793    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.116376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.116336    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"]
Nov 03 00:20:03.120142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.120113    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"]
Nov 03 00:20:03.137732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.137702    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.144344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.144312    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"]
Nov 03 00:20:03.173638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.173605    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.173812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.173662    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.173812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.173792    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.173910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.173884    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.173946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.173919    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.174088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174068    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.174192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174115    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.174192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174142    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5g9pc\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-kube-api-access-5g9pc\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.174311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174249    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.174367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174353    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.174412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174387    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.174473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174447    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.174559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174538    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmm6h\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-kube-api-access-gmm6h\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.174618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174598    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.174678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174630    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.174678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174663    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ba0b79-946a-44df-b0b3-befc1d76751f-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.174783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174695    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.174783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.174742    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276022    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276111    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276138    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276184    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276219    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmm6h\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-kube-api-access-gmm6h\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276284    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276310    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276342    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ba0b79-946a-44df-b0b3-befc1d76751f-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276375    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276399    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276437    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276475    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276519    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276579    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276601    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276614    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276727    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.276957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276774    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276778    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.276957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.276825    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5g9pc\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-kube-api-access-5g9pc\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.277364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.277312    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.277478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.277451    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.277607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.277487    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ba0b79-946a-44df-b0b3-befc1d76751f-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.277607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.277512    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.277785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.277746    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.277785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.277767    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.278219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.278195    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.278364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.278203    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.279551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.279523    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.279706 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.279688    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.280013 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.279999    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.280520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.280494    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.295788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.295765    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmm6h\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-kube-api-access-gmm6h\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.296012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.295981    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.296417 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.296398    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5g9pc\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-kube-api-access-5g9pc\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.296608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.296586    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.403965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.403933    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:03.460909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.460468    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:03.721399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.721363    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"]
Nov 03 00:20:03.722521 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:20:03.722498    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ba0b79_946a_44df_b0b3_befc1d76751f.slice/crio-b59afc8251fb91ab4decc893bc3570ea693f71e959625c35cafc55abfd781d59 WatchSource:0}: Error finding container b59afc8251fb91ab4decc893bc3570ea693f71e959625c35cafc55abfd781d59: Status 404 returned error can't find the container with id b59afc8251fb91ab4decc893bc3570ea693f71e959625c35cafc55abfd781d59
Nov 03 00:20:03.725040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.725011    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:03.725112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.725099    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:03.725155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.725132    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:03.782224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.782197    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"]
Nov 03 00:20:03.782476 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:20:03.782453    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc2ff918_fa0b_4719_b4f1_ed8e723c96fc.slice/crio-5305eaa19769694e83eca106a5c00e3b5b17b604224a623f617d49938e90d931 WatchSource:0}: Error finding container 5305eaa19769694e83eca106a5c00e3b5b17b604224a623f617d49938e90d931: Status 404 returned error can't find the container with id 5305eaa19769694e83eca106a5c00e3b5b17b604224a623f617d49938e90d931
Nov 03 00:20:03.785032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.784998    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:03.785140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.785077    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:03.785140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:03.785124    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:04.096892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.096746    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" event={"ID":"91ba0b79-946a-44df-b0b3-befc1d76751f","Type":"ContainerStarted","Data":"bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84"}
Nov 03 00:20:04.096892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.096807    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" event={"ID":"91ba0b79-946a-44df-b0b3-befc1d76751f","Type":"ContainerStarted","Data":"b59afc8251fb91ab4decc893bc3570ea693f71e959625c35cafc55abfd781d59"}
Nov 03 00:20:04.098711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.098666    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" event={"ID":"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc","Type":"ContainerStarted","Data":"b24bfdd53823c1c4e3c2460224549b70ec0aae11ee41d8108189ddfc90067729"}
Nov 03 00:20:04.098828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.098740    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" event={"ID":"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc","Type":"ContainerStarted","Data":"5305eaa19769694e83eca106a5c00e3b5b17b604224a623f617d49938e90d931"}
Nov 03 00:20:04.137556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.137497    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podStartSLOduration=1.137477035 podStartE2EDuration="1.137477035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:04.135362913 +0000 UTC m=+841.690714938" watchObservedRunningTime="2026-11-03 00:20:04.137477035 +0000 UTC m=+841.692829060"
Nov 03 00:20:04.175475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.175416    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podStartSLOduration=1.175396047 podStartE2EDuration="1.175396047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:04.173656922 +0000 UTC m=+841.729008946" watchObservedRunningTime="2026-11-03 00:20:04.175396047 +0000 UTC m=+841.730748072"
Nov 03 00:20:04.406793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.406342    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:04.406793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.406478    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:20:04.406793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.406524    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 00:20:04.461577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.461547    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:04.462627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.462607    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 00:20:04.462736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.462652    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 00:20:04.642447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:04.642404    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:20:05.408768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:05.408720    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:20:05.409327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:05.408795    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 00:20:05.461572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:05.461534    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 00:20:05.461746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:05.461596    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 00:20:06.407282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:06.404347    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:20:06.407282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:06.404412    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 00:20:06.477659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:06.477190    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 00:20:06.477659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:06.477272    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 00:20:07.409282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:07.406609    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:20:07.409282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:07.406675    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 00:20:07.460998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:07.460958    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 00:20:07.461177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:07.461021    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 00:20:08.405687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:08.405649    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:20:08.406288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:08.406229    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 00:20:08.461826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:08.461782    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 00:20:08.462058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:08.462037    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 00:20:09.407794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:09.407755    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:20:09.408474 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:09.408443    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 00:20:09.461413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:09.461378    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 00:20:09.461580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:09.461452    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 00:20:10.404435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:10.404401    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 03 00:20:10.404753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:10.404462    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 03 00:20:10.461614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:10.461575    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 03 00:20:10.462065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:10.461648    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 03 00:20:11.543758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:11.543659    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:11.564410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:11.564380    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:12.253096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:12.253063    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:12.460435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:12.460399    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:12.460624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:12.460473    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:12.503512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:12.503398    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:12.503793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:12.503749    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:13.056519 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:13.056486    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:13.057059 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:13.056919    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:13.057903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:13.057883    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:20:13.232228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:13.232197    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:13.232791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:13.232769    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:13.233691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:13.233660    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:20:18.240469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.240434    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"]
Nov 03 00:20:18.247495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.247466    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.280280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.280234    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"]
Nov 03 00:20:18.349341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349306    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349356    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349484    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349518    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349583    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h85j8\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-kube-api-access-h85j8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349611    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349696    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349727    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.349783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.349758    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.451556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451524    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.451749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451569    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.451749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451621    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.451749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451644    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.451933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451791    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h85j8\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-kube-api-access-h85j8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.451933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451844    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.451933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451927    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.452099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.451972    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.452099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.452026    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.452099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.452027    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.452245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.452025    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.452245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.452234    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.452419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.452400    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.452899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.452872    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.454148 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.454129    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.454681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.454659    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.470210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.470187    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.471451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.471433    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h85j8\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-kube-api-access-h85j8\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.563553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.563466    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:18.905972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.905939    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"]
Nov 03 00:20:18.906345 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:20:18.906302    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c43838a_640d_4fe9_81c6_f1cb1d8eb327.slice/crio-8e247e40a3f82de84f7d6c9d037f38096132a21692c50d39144174e2686bfcf9 WatchSource:0}: Error finding container 8e247e40a3f82de84f7d6c9d037f38096132a21692c50d39144174e2686bfcf9: Status 404 returned error can't find the container with id 8e247e40a3f82de84f7d6c9d037f38096132a21692c50d39144174e2686bfcf9
Nov 03 00:20:18.909172 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.909132    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:18.909282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.909222    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:18.909324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:18.909296    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:19.300505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:19.299423    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" event={"ID":"1c43838a-640d-4fe9-81c6-f1cb1d8eb327","Type":"ContainerStarted","Data":"0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf"}
Nov 03 00:20:19.300505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:19.299506    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" event={"ID":"1c43838a-640d-4fe9-81c6-f1cb1d8eb327","Type":"ContainerStarted","Data":"8e247e40a3f82de84f7d6c9d037f38096132a21692c50d39144174e2686bfcf9"}
Nov 03 00:20:19.406305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:19.399461    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podStartSLOduration=1.399438942 podStartE2EDuration="1.399438942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:19.381878734 +0000 UTC m=+856.937230783" watchObservedRunningTime="2026-11-03 00:20:19.399438942 +0000 UTC m=+856.954790970"
Nov 03 00:20:19.569350 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:19.569239    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:19.569692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:19.569665    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:19.569870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:19.569842    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:20.073813 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:20:20.073772    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:20:20.573282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:20.569703    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:20.573282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:20.569763    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:20.978299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:20.977920    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:21.240343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:21.239348    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:21.567808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:21.567710    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:21.568085 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:21.568043    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:21.689282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:21.687696    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:21.981767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:21.979605    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:20:21.981767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:21.980288    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:20:22.020616 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.019355    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.212281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.203359    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:20:22.212281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.203422    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.354364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.353523    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.354541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.354359    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:22.436368 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.432782    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:20:22.556835 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.554453    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:22.573272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.569618    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:22.573530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.573502    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:22.703328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.695897    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:22.708161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.708121    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"]
Nov 03 00:20:22.732293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.730634    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"]
Nov 03 00:20:22.739609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.738719    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:22.854522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.854494    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"kube-root-ca.crt\""
Nov 03 00:20:22.899895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.899866    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"default-dockercfg-hzjrw\""
Nov 03 00:20:22.941757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.941727    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"openshift-service-ca.crt\""
Nov 03 00:20:23.006345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:22.966432    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"distinct-w1-b85343-kserve-self-signed-certs\""
Nov 03 00:20:23.291854 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.291730    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tls-certs\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.306279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.292353    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tmp-dir\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.321777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.321741    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-dshm\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.322044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.322019    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fmhh\" (UniqueName: \"kubernetes.io/projected/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-kube-api-access-9fmhh\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.366640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.366603    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-model-cache\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.367035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.367008    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-home\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.467969    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tls-certs\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.468072    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tmp-dir\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.468109    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-dshm\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.468129    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fmhh\" (UniqueName: \"kubernetes.io/projected/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-kube-api-access-9fmhh\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.468173    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-model-cache\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.468213    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-home\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.468735    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tmp-dir\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.468937 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.468905    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-home\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.469185 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.469110    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-model-cache\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.471142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.471115    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-dshm\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.471521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.471505    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tls-certs\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.536981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.536947    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fmhh\" (UniqueName: \"kubernetes.io/projected/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-kube-api-access-9fmhh\") pod \"distinct-w1-b85343-kserve-9d8dd6cf9-4rptl\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:23.564394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.564312    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:23.564394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.564364    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:23.779334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:23.779300    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:24.224093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:24.224053    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"]
Nov 03 00:20:24.225128 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:20:24.225103    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd460a3cf_16dd_49ad_a3d8_05ab4a8112ca.slice/crio-7d5c720eb70f13cb5d1e57b4594a57e025b1158438cc6d916d533ba01e7ca096 WatchSource:0}: Error finding container 7d5c720eb70f13cb5d1e57b4594a57e025b1158438cc6d916d533ba01e7ca096: Status 404 returned error can't find the container with id 7d5c720eb70f13cb5d1e57b4594a57e025b1158438cc6d916d533ba01e7ca096
Nov 03 00:20:24.473496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:24.473461    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" event={"ID":"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca","Type":"ContainerStarted","Data":"62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193"}
Nov 03 00:20:24.473699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:24.473509    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" event={"ID":"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca","Type":"ContainerStarted","Data":"7d5c720eb70f13cb5d1e57b4594a57e025b1158438cc6d916d533ba01e7ca096"}
Nov 03 00:20:24.540056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:24.539956    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podStartSLOduration=2.539942423 podStartE2EDuration="2.539942423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:24.537474768 +0000 UTC m=+862.092826791" watchObservedRunningTime="2026-11-03 00:20:24.539942423 +0000 UTC m=+862.095294446"
Nov 03 00:20:24.564404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:24.564371    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:24.564561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:24.564429    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:24.779640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:24.779605    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:25.564641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:25.564606    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:25.564852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:25.564665    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:26.564225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:26.564189    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:26.564666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:26.564245    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:27.564000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:27.563962    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 03 00:20:27.564211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:27.564019    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 03 00:20:28.569000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:28.568968    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:28.569478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:28.569371    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:28.570056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:28.570038    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:20:31.130447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:31.130414    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:20:33.291281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.283541    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"]
Nov 03 00:20:33.314281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.309344    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.446369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.434316    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"]
Nov 03 00:20:33.483044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.447388    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.497800 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.497762    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.498076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.498053    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.498234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.498218    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.498393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.498372    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4c8zr\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-kube-api-access-4c8zr\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.498725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.498706    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.498891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.498876    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.499045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.499028    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.499156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.499142    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.621410    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.622954    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623003    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623031    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623073    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623101    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4c8zr\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-kube-api-access-4c8zr\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623292    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623351    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623393    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.639281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.623419    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.680277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.679330    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:33.705282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.697037    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.705282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.697758    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.730281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.706993    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.730281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.707302    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.730281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.707756    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.730281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.708274    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.763929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.763863    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.789507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.779480    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:33.818326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.802066    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4c8zr\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-kube-api-access-4c8zr\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:33.818326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:33.815783    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:34.084333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:34.083502    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:34.437283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:34.436775    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:20:34.441279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:34.439378    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:34.631288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:34.623400    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:34.736770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:34.736661    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:34.827058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:34.803451    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:34.827058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:34.803546    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:20:35.334337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.334301    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"]
Nov 03 00:20:35.482499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.482457    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:35.507044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.507015    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:20:35.507370 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.507350    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:20:35.507537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.507523    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:20:35.563018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.562984    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:20:35.743936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.734238    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:35.743936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.734369    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:35.743936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.734413    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:35.798337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.798306    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:20:35.884874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:35.884794    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" event={"ID":"dda4f231-46f1-4753-bb41-d5e8745dd7d5","Type":"ContainerStarted","Data":"c15bba0c1b6049e5df6ea24af3f29ee6a561aa9bb5483dd8cb67280d4eb3eb26"}
Nov 03 00:20:36.916400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:36.916367    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" event={"ID":"dda4f231-46f1-4753-bb41-d5e8745dd7d5","Type":"ContainerStarted","Data":"7f46da08973d0c0b2e733f64d0bda577f692f7ec49ec23b460ecd5d681bc16c6"}
Nov 03 00:20:36.969433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:36.969380    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podStartSLOduration=3.969366914 podStartE2EDuration="3.969366914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:36.966950863 +0000 UTC m=+874.522302887" watchObservedRunningTime="2026-11-03 00:20:36.969366914 +0000 UTC m=+874.524719018"
Nov 03 00:20:37.111280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:37.105682    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:37.112868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:37.112750    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:20:37.112868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:37.112821    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 00:20:38.095729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:38.095690    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:20:38.111923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:38.111461    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 00:20:39.088732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:39.088694    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:20:39.089030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:39.088988    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 00:20:40.090742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:40.090703    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:20:40.091391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:40.091354    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 00:20:41.093680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:41.093643    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:20:41.094305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:41.094271    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 00:20:42.099346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:42.097658    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Nov 03 00:20:42.099346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:42.097720    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Nov 03 00:20:43.518080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:43.518046    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:43.518592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:43.518482    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:43.519427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:43.519404    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 00:20:48.864585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.864551    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"]
Nov 03 00:20:48.877469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.877441    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"]
Nov 03 00:20:48.888475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.888393    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.952774    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.952875    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczxd\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-kube-api-access-wczxd\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.952913    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.952945    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.952988    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.953030    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.953108    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797b5565-9c2a-4dbd-9f73-20865f557797-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.953135    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:48.961279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:48.953210    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797b5565-9c2a-4dbd-9f73-20865f557797-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.057068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.057025    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.058481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.058453    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczxd\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-kube-api-access-wczxd\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.058671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.058652    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.058808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.058792    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.058962 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.058944    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.059109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.059093    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.059307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.059288    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797b5565-9c2a-4dbd-9f73-20865f557797-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.059435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.059420    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.059606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.059590    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797b5565-9c2a-4dbd-9f73-20865f557797-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.060594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.057717    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.072355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.061133    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.097764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.061401    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.097980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.061630    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.098147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.098051    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczxd\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-kube-api-access-wczxd\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.098147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.082407    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.098147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.061996    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797b5565-9c2a-4dbd-9f73-20865f557797-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.129327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.120859    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797b5565-9c2a-4dbd-9f73-20865f557797-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.129967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.129938    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.331658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.331624    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:49.526521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.526392    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:49.589373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.589331    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 00:20:49.824873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:49.824831    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:50.299545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:50.299499    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"]
Nov 03 00:20:50.302310 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:50.302240    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:50.302885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:50.302859    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:50.303527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:50.303489    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:20:50.569034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:50.539472    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:20:50.569034 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:20:50.540945    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:20:51.249788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:51.249749    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" event={"ID":"797b5565-9c2a-4dbd-9f73-20865f557797","Type":"ContainerStarted","Data":"f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02"}
Nov 03 00:20:51.249970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:51.249805    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" event={"ID":"797b5565-9c2a-4dbd-9f73-20865f557797","Type":"ContainerStarted","Data":"ef596565d65bdb1bb3261bf1d0126b557cb1870f8c4530bd0755b33bb83b4278"}
Nov 03 00:20:51.359794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:51.324427    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podStartSLOduration=3.324404645 podStartE2EDuration="3.324404645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:51.310829493 +0000 UTC m=+888.866181530" watchObservedRunningTime="2026-11-03 00:20:51.324404645 +0000 UTC m=+888.879756663"
Nov 03 00:20:51.370711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:51.367326    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:51.371119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:51.371023    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 00:20:51.372315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:51.371438    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 00:20:52.333368 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:52.333335    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 00:20:52.333552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:52.333385    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 00:20:53.332866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:53.332828    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 00:20:53.333393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:53.332889    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 00:20:54.333327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:54.333294    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 00:20:54.333750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:54.333348    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 00:20:55.335358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:55.332974    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 00:20:55.335358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:55.333044    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 00:20:56.333454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:56.333417    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 00:20:56.333661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:56.333471    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 00:20:57.271743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.271698    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"]
Nov 03 00:20:57.277362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.277333    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.283570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.283544    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"]
Nov 03 00:20:57.291834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.291799    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-b85343\"/\"distinct-2-w1-b85343-kserve-self-signed-certs\""
Nov 03 00:20:57.333340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.333300    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 03 00:20:57.333497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.333368    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 03 00:20:57.369617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.369592    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-tmp-dir\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.369767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.369639    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-model-cache\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.369767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.369658    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-dshm\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.369767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.369759    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6554\" (UniqueName: \"kubernetes.io/projected/ea440629-1449-4a15-8565-304fb54877ce-kube-api-access-p6554\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.369927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.369780    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-home\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.369966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.369943    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440629-1449-4a15-8565-304fb54877ce-tls-certs\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.480549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.480511    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-tmp-dir\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.480942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.480920    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-model-cache\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.481079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.481065    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-dshm\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.481249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.481217    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6554\" (UniqueName: \"kubernetes.io/projected/ea440629-1449-4a15-8565-304fb54877ce-kube-api-access-p6554\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.481450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.481432    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-home\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.481818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.481785    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440629-1449-4a15-8565-304fb54877ce-tls-certs\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.483166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.483131    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-tmp-dir\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.495437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.489692    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-model-cache\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.495631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.489873    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-home\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.550324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.515579    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-dshm\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.550324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.527875    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440629-1449-4a15-8565-304fb54877ce-tls-certs\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.556338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.553958    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6554\" (UniqueName: \"kubernetes.io/projected/ea440629-1449-4a15-8565-304fb54877ce-kube-api-access-p6554\") pod \"distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") " pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.593104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.593069    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:57.901663 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:20:57.901632    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea440629_1449_4a15_8565_304fb54877ce.slice/crio-5af684ea7d39b7fbe994f0322cd0049a6fceeaf0e04bb66a002b30acebfcc5d1 WatchSource:0}: Error finding container 5af684ea7d39b7fbe994f0322cd0049a6fceeaf0e04bb66a002b30acebfcc5d1: Status 404 returned error can't find the container with id 5af684ea7d39b7fbe994f0322cd0049a6fceeaf0e04bb66a002b30acebfcc5d1
Nov 03 00:20:57.903462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:57.903435    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"]
Nov 03 00:20:58.295137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:58.295049    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" event={"ID":"ea440629-1449-4a15-8565-304fb54877ce","Type":"ContainerStarted","Data":"0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89"}
Nov 03 00:20:58.295137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:58.295111    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" event={"ID":"ea440629-1449-4a15-8565-304fb54877ce","Type":"ContainerStarted","Data":"5af684ea7d39b7fbe994f0322cd0049a6fceeaf0e04bb66a002b30acebfcc5d1"}
Nov 03 00:20:58.333060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:58.332996    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podStartSLOduration=1.332968898 podStartE2EDuration="1.332968898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:20:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:20:58.33129943 +0000 UTC m=+895.886651454" watchObservedRunningTime="2026-11-03 00:20:58.332968898 +0000 UTC m=+895.888320923"
Nov 03 00:20:58.337994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:58.337965    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:58.338435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:58.338410    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:58.339356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:58.339333    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:20:58.606684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:58.601822    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:20:59.281299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:20:59.281241    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:21:03.377563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.377528    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"]
Nov 03 00:21:03.383024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.383003    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.416843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.416807    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"]
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422105    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422150    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422185    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422345    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nsd2\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-kube-api-access-6nsd2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422378    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422434    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422466    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422559    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.424608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.422602    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.524573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.524532    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.524573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.524574    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.524822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.524686    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.524890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.524849    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nsd2\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-kube-api-access-6nsd2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.524946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.524894    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.524946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.524910    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.524963    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.525000    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.525012    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.525131    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.525184    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.525218    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.525503    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.525811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.525784    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.527214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.527192    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.527327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.527281    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.544019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.543981    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.544151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.544136    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nsd2\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-kube-api-access-6nsd2\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:03.698696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:03.698596    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:04.018747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.018718    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"]
Nov 03 00:21:04.021911 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:21:04.021885    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402e4e66_b1a9_448b_94b5_1da2ff1ca5a5.slice/crio-5f6a8e4b5698254ace44022a5c1626f1ccce2fb89ad88b9fa4af7a4172995612 WatchSource:0}: Error finding container 5f6a8e4b5698254ace44022a5c1626f1ccce2fb89ad88b9fa4af7a4172995612: Status 404 returned error can't find the container with id 5f6a8e4b5698254ace44022a5c1626f1ccce2fb89ad88b9fa4af7a4172995612
Nov 03 00:21:04.024707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.024668    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:21:04.024805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.024760    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:21:04.024805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.024798    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:21:04.378096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.378023    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" event={"ID":"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5","Type":"ContainerStarted","Data":"7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548"}
Nov 03 00:21:04.378613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.378127    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" event={"ID":"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5","Type":"ContainerStarted","Data":"5f6a8e4b5698254ace44022a5c1626f1ccce2fb89ad88b9fa4af7a4172995612"}
Nov 03 00:21:04.440720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.440675    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podStartSLOduration=1.440663247 podStartE2EDuration="1.440663247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:04.428727614 +0000 UTC m=+901.984079639" watchObservedRunningTime="2026-11-03 00:21:04.440663247 +0000 UTC m=+901.996015271"
Nov 03 00:21:04.699379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:04.699214    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:05.709281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:05.703054    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:21:05.709281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:05.703122    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 00:21:06.710283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:06.707356    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:21:06.710283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:06.707429    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 00:21:07.598290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:07.597917    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:21:07.624114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:07.624081    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:21:07.707282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:07.699807    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:21:07.711436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:07.711389    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 00:21:08.463162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:08.463129    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:21:08.711438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:08.711391    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:21:08.711882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:08.711476    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 00:21:09.706654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:09.706615    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:21:09.708118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:09.706903    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 00:21:10.698939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:10.698903    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 03 00:21:10.699393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:10.698977    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 03 00:21:11.703826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:11.703794    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:11.704250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:11.704084    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:11.704771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:11.704752    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:21:18.071430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:18.071377    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:18.456823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:18.456778    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 00:21:18.537961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:18.527397    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 00:21:18.537961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:18.535648    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:21:18.600633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:18.600573    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 00:21:18.688925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:18.678351    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:19.905764 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:21:19.905726    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:21:26.122862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:26.122827    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:21:29.482281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:29.460549    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.526280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:29.512429    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.526280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:29.512637    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.590286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:29.588396    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:29.590286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:29.588493    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:29.590286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:29.589427    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 00:21:29.799654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:29.799554    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.202323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.185818    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.723580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.720346    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.789283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.788877    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.808421 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.808371    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:30.812881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.812854    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:21:30.849281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.840429    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:21:30.849281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.813313    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:21:30.854292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:30.834689    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.233526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.233482    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.266381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.256889    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:21:31.266381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.256958    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:21:31.266381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.257000    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:21:31.266381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.257034    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:21:31.727281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.726404    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:21:31.752283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.727981    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.835079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.809352    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.852980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.848072    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:31.853670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:31.853638    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:32.237815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:32.237233    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:32.266903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:32.266860    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:21:32.294802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:32.294777    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:21:32.302745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:32.302722    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:21:32.344024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:32.343983    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:21:32.349905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:32.349873    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:21:38.404778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.404743    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"]
Nov 03 00:21:38.410199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.410176    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:38.428828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.428800    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"]
Nov 03 00:21:38.436685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.436654    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9f76b708\""
Nov 03 00:21:38.436685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.436663    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9f76b708-dockercfg-4pkrc\""
Nov 03 00:21:38.491545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.491514    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzl5g\" (UniqueName: \"kubernetes.io/projected/c53373dc-5938-4bc0-946d-4b064888f165-kube-api-access-wzl5g\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:38.491713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.491669    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53373dc-5938-4bc0-946d-4b064888f165-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:38.701325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.700988    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzl5g\" (UniqueName: \"kubernetes.io/projected/c53373dc-5938-4bc0-946d-4b064888f165-kube-api-access-wzl5g\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:38.701325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.701199    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53373dc-5938-4bc0-946d-4b064888f165-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:38.706213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.706171    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53373dc-5938-4bc0-946d-4b064888f165-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:38.748314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:38.746656    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzl5g\" (UniqueName: \"kubernetes.io/projected/c53373dc-5938-4bc0-946d-4b064888f165-kube-api-access-wzl5g\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:39.024454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:39.024370    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:39.328661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:39.328629    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"]
Nov 03 00:21:39.329735 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:21:39.329711    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53373dc_5938_4bc0_946d_4b064888f165.slice/crio-7f6e66257a3fc640fe99d20117d0c15465a9c2c4a1aaecfccfd1318d8b8d985f WatchSource:0}: Error finding container 7f6e66257a3fc640fe99d20117d0c15465a9c2c4a1aaecfccfd1318d8b8d985f: Status 404 returned error can't find the container with id 7f6e66257a3fc640fe99d20117d0c15465a9c2c4a1aaecfccfd1318d8b8d985f
Nov 03 00:21:39.789826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:39.789785    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" event={"ID":"c53373dc-5938-4bc0-946d-4b064888f165","Type":"ContainerStarted","Data":"f70c4b0a97b96ec319e1793ee9bdf9be59be76b1245e557d4943b192ef1bfd4c"}
Nov 03 00:21:39.790324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:39.789852    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" event={"ID":"c53373dc-5938-4bc0-946d-4b064888f165","Type":"ContainerStarted","Data":"7f6e66257a3fc640fe99d20117d0c15465a9c2c4a1aaecfccfd1318d8b8d985f"}
Nov 03 00:21:39.790324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:39.789883    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:39.830148 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:39.830097    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" podStartSLOduration=1.830082575 podStartE2EDuration="1.830082575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:39.825839069 +0000 UTC m=+937.381191093" watchObservedRunningTime="2026-11-03 00:21:39.830082575 +0000 UTC m=+937.385434600"
Nov 03 00:21:44.277281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.275436    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.298278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.284349    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:44.298278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.284420    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.360280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.359387    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.563339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.562415    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:21:44.563339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.562497    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.800866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.782997    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:21:44.800866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:44.783495    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 00:21:48.272607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.272572    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"]
Nov 03 00:21:48.276846 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.276826    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.296029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.296002    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"]
Nov 03 00:21:48.304926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.304318    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9f76b708-serving-cert\""
Nov 03 00:21:48.432479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.432440    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcwff\" (UniqueName: \"kubernetes.io/projected/90423226-4361-4905-9ac2-f1850aaa0996-kube-api-access-jcwff\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.432649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.432486    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90423226-4361-4905-9ac2-f1850aaa0996-openshift-service-ca\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.432649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.432506    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-maas-api-tls\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.432732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.432713    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-metrics-certs\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.452427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.452393    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"]
Nov 03 00:21:48.494557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.494523    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.521281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.515175    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"]
Nov 03 00:21:48.584193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.584103    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcwff\" (UniqueName: \"kubernetes.io/projected/90423226-4361-4905-9ac2-f1850aaa0996-kube-api-access-jcwff\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.584795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.584761    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.585367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.585347    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90423226-4361-4905-9ac2-f1850aaa0996-openshift-service-ca\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.595578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595550    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-maas-api-tls\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.595702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.586772    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90423226-4361-4905-9ac2-f1850aaa0996-openshift-service-ca\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.595702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595591    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.595702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595621    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7a4329-145a-41c8-bc2d-97dd98a8d113-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.595892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595739    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.595892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595784    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.595892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595804    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzfng\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-kube-api-access-tzfng\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.595892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595841    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-metrics-certs\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.596112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595913    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.596112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.595970    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.596112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.596001    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.601329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.600120    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-maas-api-tls\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.609728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.609705    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-metrics-certs\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.616102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.616080    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcwff\" (UniqueName: \"kubernetes.io/projected/90423226-4361-4905-9ac2-f1850aaa0996-kube-api-access-jcwff\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-m468c\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.696769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.696738    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.696948 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.696784    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.696948 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.696896    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697033    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697085    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697109    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7a4329-145a-41c8-bc2d-97dd98a8d113-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697229    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697291    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697310    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697341    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzfng\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-kube-api-access-tzfng\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697446    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697619    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697820    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.697900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.697882    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7a4329-145a-41c8-bc2d-97dd98a8d113-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.699543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.699524    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.699647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.699590    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.737175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.737142    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.738058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.738024    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzfng\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-kube-api-access-tzfng\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:48.898619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.898580    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:48.899484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:48.899452    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:49.217679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.217648    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"]
Nov 03 00:21:49.218481 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:21:49.218442    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90423226_4361_4905_9ac2_f1850aaa0996.slice/crio-a81292e809d282fa31cb4a613fbafdf424ce914c8b68f9b3b639d9c4e92e08b8 WatchSource:0}: Error finding container a81292e809d282fa31cb4a613fbafdf424ce914c8b68f9b3b639d9c4e92e08b8: Status 404 returned error can't find the container with id a81292e809d282fa31cb4a613fbafdf424ce914c8b68f9b3b639d9c4e92e08b8
Nov 03 00:21:49.220033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.220016    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:21:49.223285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:21:49.223245    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf7a4329_145a_41c8_bc2d_97dd98a8d113.slice/crio-52c8e690d0b5b9fb792218c47b02624a0926ae1547c55c302b72b4e46de75c07 WatchSource:0}: Error finding container 52c8e690d0b5b9fb792218c47b02624a0926ae1547c55c302b72b4e46de75c07: Status 404 returned error can't find the container with id 52c8e690d0b5b9fb792218c47b02624a0926ae1547c55c302b72b4e46de75c07
Nov 03 00:21:49.225671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.225638    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:21:49.225820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.225701    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:21:49.225820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.225729    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:21:49.228964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.228944    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"]
Nov 03 00:21:49.947533 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:21:49.947492    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:21:49.951123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.951094    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" event={"ID":"df7a4329-145a-41c8-bc2d-97dd98a8d113","Type":"ContainerStarted","Data":"8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91"}
Nov 03 00:21:49.951379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.951354    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" event={"ID":"df7a4329-145a-41c8-bc2d-97dd98a8d113","Type":"ContainerStarted","Data":"52c8e690d0b5b9fb792218c47b02624a0926ae1547c55c302b72b4e46de75c07"}
Nov 03 00:21:49.952478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:49.952456    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c" event={"ID":"90423226-4361-4905-9ac2-f1850aaa0996","Type":"ContainerStarted","Data":"a81292e809d282fa31cb4a613fbafdf424ce914c8b68f9b3b639d9c4e92e08b8"}
Nov 03 00:21:50.017679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:50.017611    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podStartSLOduration=2.017589491 podStartE2EDuration="2.017589491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:50.01578223 +0000 UTC m=+947.571134271" watchObservedRunningTime="2026-11-03 00:21:50.017589491 +0000 UTC m=+947.572941516"
Nov 03 00:21:50.812045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:50.812004    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:21:50.912319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:50.907347    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:50.912319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:50.907503    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:21:50.912319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:50.907545    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 00:21:50.967793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:50.965110    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c" event={"ID":"90423226-4361-4905-9ac2-f1850aaa0996","Type":"ContainerStarted","Data":"6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013"}
Nov 03 00:21:50.967793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:50.966805    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:21:51.909280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:51.908646    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:21:51.909280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:51.908710    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 00:21:52.906282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:52.902618    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:21:52.906282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:52.902682    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 00:21:53.918734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:53.918685    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:21:53.919312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:53.918746    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 00:21:54.904656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:54.904616    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:21:54.904970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:54.904939    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 00:21:55.899892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:55.899855    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body=
Nov 03 00:21:55.900324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:55.899925    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused"
Nov 03 00:21:56.905417 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:56.905308    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:56.906099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:56.906044    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:56.907365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:56.907345    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:21:56.951477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:56.951419    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c" podStartSLOduration=8.337202036 podStartE2EDuration="8.951404793s" totalImagesPullingTime="614.202757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:21:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:21:51.012745148 +0000 UTC m=+948.568097176" watchObservedRunningTime="2026-11-03 00:21:56.951404793 +0000 UTC m=+954.506756835"
Nov 03 00:21:57.981810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:21:57.981781    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:22:01.556296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:01.556247    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:22:03.540609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.540573    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"]
Nov 03 00:22:03.549108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.549070    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.563575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.563549    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"]
Nov 03 00:22:03.567701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.567680    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.581018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.580992    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"]
Nov 03 00:22:03.605921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.604579    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"]
Nov 03 00:22:03.617811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.617778    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.618061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.618037    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.618193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.618176    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.618361 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.618340    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.618500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.618483    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.618640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.618623    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.618751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.618736    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ceba1e-6211-4524-9062-037a85917c22-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.618941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.618924    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.619128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619096    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ceba1e-6211-4524-9062-037a85917c22-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.619366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619343    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.619467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619386    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.619467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619416    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.619467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619445    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.619645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619475    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkr9f\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-kube-api-access-rkr9f\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.619645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619501    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmzzt\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-kube-api-access-pmzzt\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.619645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619556    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.619645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619580    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.619645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.619613    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.720800 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.720751    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.720994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.720880    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.720994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.720949    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.720994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.720975    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.721192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721022    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.721192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721060    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.721192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721094    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.721192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721123    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ceba1e-6211-4524-9062-037a85917c22-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.721476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721213    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.721476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721246    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ceba1e-6211-4524-9062-037a85917c22-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.721476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721342    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.721476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721378    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.721476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721386    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.721476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721386    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.721806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721704    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.721806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721722    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.721919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721904    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.721972    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722015    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ceba1e-6211-4524-9062-037a85917c22-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722061    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722095    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722131    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkr9f\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-kube-api-access-rkr9f\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722161    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmzzt\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-kube-api-access-pmzzt\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722215    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722224    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722238    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722304    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.722784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.722607    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.725199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.724834    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.725199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.725042    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.738293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.726530    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.738293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.726662    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ceba1e-6211-4524-9062-037a85917c22-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.743674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.743640    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmzzt\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-kube-api-access-pmzzt\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.744637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.744615    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.745975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.745928    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:03.748587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.747781    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkr9f\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-kube-api-access-rkr9f\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.870344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.870309    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:03.882960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:03.882934    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:04.246052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.245979    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"]
Nov 03 00:22:04.250530 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:22:04.250496    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod529a05c4_4cd7_4729_bcb9_70cfa9b6d448.slice/crio-bae13a46614dc3f26c21ffb4a393074f891beed3470941dcd3fb0ec986f61b33 WatchSource:0}: Error finding container bae13a46614dc3f26c21ffb4a393074f891beed3470941dcd3fb0ec986f61b33: Status 404 returned error can't find the container with id bae13a46614dc3f26c21ffb4a393074f891beed3470941dcd3fb0ec986f61b33
Nov 03 00:22:04.253413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.253373    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:04.253509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.253466    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:04.253509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.253502    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:04.290208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.290180    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"]
Nov 03 00:22:04.292979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.292938    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:04.293086 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.293027    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:04.293086 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:04.293070    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:05.076496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.076462    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" event={"ID":"35ceba1e-6211-4524-9062-037a85917c22","Type":"ContainerStarted","Data":"bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764"}
Nov 03 00:22:05.076971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.076523    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" event={"ID":"35ceba1e-6211-4524-9062-037a85917c22","Type":"ContainerStarted","Data":"e0126dde8bf17fd1aaf940319f5f6b89ffc18c1f3e6358e03187dba1984ec7e4"}
Nov 03 00:22:05.078290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.078243    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" event={"ID":"529a05c4-4cd7-4729-bcb9-70cfa9b6d448","Type":"ContainerStarted","Data":"a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605"}
Nov 03 00:22:05.078404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.078295    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" event={"ID":"529a05c4-4cd7-4729-bcb9-70cfa9b6d448","Type":"ContainerStarted","Data":"bae13a46614dc3f26c21ffb4a393074f891beed3470941dcd3fb0ec986f61b33"}
Nov 03 00:22:05.159620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.159562    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podStartSLOduration=2.159546163 podStartE2EDuration="2.159546163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:05.145848675 +0000 UTC m=+962.701200699" watchObservedRunningTime="2026-11-03 00:22:05.159546163 +0000 UTC m=+962.714898188"
Nov 03 00:22:05.188294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.186690    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podStartSLOduration=2.186672069 podStartE2EDuration="2.186672069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:05.184577558 +0000 UTC m=+962.739929583" watchObservedRunningTime="2026-11-03 00:22:05.186672069 +0000 UTC m=+962.742024095"
Nov 03 00:22:05.870793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.870761    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:05.871996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.871974    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:05.872092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.872021    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 00:22:05.883346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.883321    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:05.884535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.884510    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:05.884649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:05.884553    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 00:22:06.876560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:06.876523    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:06.877200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:06.877167    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 00:22:06.883399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:06.883369    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:06.883674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:06.883619    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 00:22:07.871517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:07.871477    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:07.871707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:07.871535    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 00:22:07.883851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:07.883804    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:07.884355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:07.883869    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 00:22:08.886570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:08.886533    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:08.887186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:08.887154    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 00:22:08.888012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:08.887990    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:08.888175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:08.888156    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 00:22:09.871427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:09.871391    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:09.871600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:09.871455    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 00:22:09.884099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:09.884073    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:09.884232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:09.884130    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 00:22:10.871465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:10.871427    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 03 00:22:10.871869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:10.871494    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 03 00:22:10.883451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:10.883419    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body=
Nov 03 00:22:10.883625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:10.883481    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused"
Nov 03 00:22:11.997215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:11.997185    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:11.997745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:11.997530    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:11.998516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:11.998496    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:22:12.000524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:12.000505    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:12.000809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:12.000787    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:12.002115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:12.002097    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:22:19.966366 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:22:19.963975    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:22:26.361767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:26.361727    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:22:29.274282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:29.274131    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.309282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:29.304340    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.360285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:29.359335    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:29.362289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:29.360535    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:30.056292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:29.988208    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:22:31.027280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.019943    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"]
Nov 03 00:22:31.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.113842    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"]
Nov 03 00:22:31.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.113972    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.159272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.158964    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-8fd232-openshift-default-dockercfg-tn46p\""
Nov 03 00:22:31.237280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.236699    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-certs\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.237280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.236783    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdj6m\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-kube-api-access-qdj6m\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.260357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.236874    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-socket\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.260357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.252688    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-podinfo\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.260357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.252766    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-token\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.260357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.252814    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffad482-272b-472c-aaa8-0641a2c7d24f-istiod-ca-cert\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.260357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.252887    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-envoy\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.260357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.252988    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-credential-socket\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.260357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.253013    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-data\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.432721    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-token\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.432785    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffad482-272b-472c-aaa8-0641a2c7d24f-istiod-ca-cert\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.432852    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-envoy\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.432936    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-credential-socket\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.432957    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-data\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.433110    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-certs\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.433144    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdj6m\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-kube-api-access-qdj6m\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.433206    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-socket\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.441280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.433237    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-podinfo\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.463331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.442588    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-credential-socket\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.463331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.450061    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-data\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.463331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.450330    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-certs\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.463331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.450763    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-socket\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.487601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.487560    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffad482-272b-472c-aaa8-0641a2c7d24f-istiod-ca-cert\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.500297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.496384    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-podinfo\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.513291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.511128    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-envoy\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.667174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.667134    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdj6m\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-kube-api-access-qdj6m\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.699849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.699773    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-token\") pod \"e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") " pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:31.874667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:31.874624    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:32.577641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:32.577293    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"]
Nov 03 00:22:32.579877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:32.579834    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:32.581570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:32.580100    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:32.581570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:32.580227    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:32.650295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:32.638026    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" event={"ID":"6ffad482-272b-472c-aaa8-0641a2c7d24f","Type":"ContainerStarted","Data":"12600acc8f668dfa590ee9b86d66a91c6fd25516f1c1c77f22763294c1f60ff9"}
Nov 03 00:22:33.645929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:33.645898    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" event={"ID":"6ffad482-272b-472c-aaa8-0641a2c7d24f","Type":"ContainerStarted","Data":"731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475"}
Nov 03 00:22:33.711501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:33.711432    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" podStartSLOduration=3.711407556 podStartE2EDuration="3.711407556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:33.695160895 +0000 UTC m=+991.250512918" watchObservedRunningTime="2026-11-03 00:22:33.711407556 +0000 UTC m=+991.266759585"
Nov 03 00:22:33.882016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:33.880711    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:33.908452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:33.908376    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:34.652268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:34.652213    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:34.653302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:34.653279    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:22:37.923116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.923081    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"]
Nov 03 00:22:37.928695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.928674    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:37.957437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.957413    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-8fd232-dockercfg-64j4q\""
Nov 03 00:22:37.957594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.957412    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-8fd232\""
Nov 03 00:22:37.961971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.961945    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"]
Nov 03 00:22:37.971166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.971111    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee1980-21d0-4d91-bc30-eae642827307-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") " pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:37.971610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.971580    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gtczm\" (UniqueName: \"kubernetes.io/projected/c8ee1980-21d0-4d91-bc30-eae642827307-kube-api-access-gtczm\") pod \"payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") " pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:37.975717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.975696    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"]
Nov 03 00:22:37.983527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.983492    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:37.999866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:37.999841    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"]
Nov 03 00:22:38.155292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.104195    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gtczm\" (UniqueName: \"kubernetes.io/projected/c8ee1980-21d0-4d91-bc30-eae642827307-kube-api-access-gtczm\") pod \"payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") " pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:38.175327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.160446    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee1980-21d0-4d91-bc30-eae642827307-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") " pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:38.175327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.161808    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee1980-21d0-4d91-bc30-eae642827307-plugins-config-volume\") pod \"payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") " pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:38.246277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.201059    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gtczm\" (UniqueName: \"kubernetes.io/projected/c8ee1980-21d0-4d91-bc30-eae642827307-kube-api-access-gtczm\") pod \"payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") " pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:38.247037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.247000    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:38.268280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.264153    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twmbp\" (UniqueName: \"kubernetes.io/projected/45ce9835-2515-423a-adeb-04f07c324e63-kube-api-access-twmbp\") pod \"payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:38.268280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.264881    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ce9835-2515-423a-adeb-04f07c324e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:38.366496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.366468    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twmbp\" (UniqueName: \"kubernetes.io/projected/45ce9835-2515-423a-adeb-04f07c324e63-kube-api-access-twmbp\") pod \"payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:38.366601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.366516    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ce9835-2515-423a-adeb-04f07c324e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:38.367748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.367717    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ce9835-2515-423a-adeb-04f07c324e63-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:38.389589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.389556    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twmbp\" (UniqueName: \"kubernetes.io/projected/45ce9835-2515-423a-adeb-04f07c324e63-kube-api-access-twmbp\") pod \"payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:38.606962 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.606932    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:38.610770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.610743    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"]
Nov 03 00:22:38.612033 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:22:38.612008    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8ee1980_21d0_4d91_bc30_eae642827307.slice/crio-6a0d6f9bda54f31861f14c88c22a2d3cfe3af70766dc4167c63c673775732a09 WatchSource:0}: Error finding container 6a0d6f9bda54f31861f14c88c22a2d3cfe3af70766dc4167c63c673775732a09: Status 404 returned error can't find the container with id 6a0d6f9bda54f31861f14c88c22a2d3cfe3af70766dc4167c63c673775732a09
Nov 03 00:22:38.692666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.692636    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk" event={"ID":"c8ee1980-21d0-4d91-bc30-eae642827307","Type":"ContainerStarted","Data":"6a0d6f9bda54f31861f14c88c22a2d3cfe3af70766dc4167c63c673775732a09"}
Nov 03 00:22:38.911683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:38.911656    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"]
Nov 03 00:22:38.911945 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:22:38.911923    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45ce9835_2515_423a_adeb_04f07c324e63.slice/crio-117de34c98118d87d67fa3faf1d48f67a95b31454cec0fcc710ad6212bc00e02 WatchSource:0}: Error finding container 117de34c98118d87d67fa3faf1d48f67a95b31454cec0fcc710ad6212bc00e02: Status 404 returned error can't find the container with id 117de34c98118d87d67fa3faf1d48f67a95b31454cec0fcc710ad6212bc00e02
Nov 03 00:22:39.700020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:39.699984    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn" event={"ID":"45ce9835-2515-423a-adeb-04f07c324e63","Type":"ContainerStarted","Data":"fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43"}
Nov 03 00:22:39.700545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:39.700040    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn" event={"ID":"45ce9835-2515-423a-adeb-04f07c324e63","Type":"ContainerStarted","Data":"117de34c98118d87d67fa3faf1d48f67a95b31454cec0fcc710ad6212bc00e02"}
Nov 03 00:22:39.700545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:39.700062    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:39.701725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:39.701695    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk" event={"ID":"c8ee1980-21d0-4d91-bc30-eae642827307","Type":"ContainerStarted","Data":"e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813"}
Nov 03 00:22:39.701844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:39.701787    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:39.738087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:39.738043    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn" podStartSLOduration=2.73803013 podStartE2EDuration="2.73803013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:39.735048609 +0000 UTC m=+997.290400633" watchObservedRunningTime="2026-11-03 00:22:39.73803013 +0000 UTC m=+997.293382154"
Nov 03 00:22:39.771205 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:39.771161    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk" podStartSLOduration=2.77114936 podStartE2EDuration="2.77114936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:39.770489645 +0000 UTC m=+997.325841919" watchObservedRunningTime="2026-11-03 00:22:39.77114936 +0000 UTC m=+997.326501384"
Nov 03 00:22:47.605301 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.605188    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"]
Nov 03 00:22:47.647280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.641422    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"]
Nov 03 00:22:47.647280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.641550    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.662093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.662069    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-8fd232-serving-cert\""
Nov 03 00:22:47.685617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.685583    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-maas-api-tls\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.685773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.685627    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkqvr\" (UniqueName: \"kubernetes.io/projected/03499319-d633-4824-a994-f8390fbaf20e-kube-api-access-gkqvr\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.685773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.685697    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-metrics-certs\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.685972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.685955    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03499319-d633-4824-a994-f8390fbaf20e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.786998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.786963    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-maas-api-tls\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.786998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.786999    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkqvr\" (UniqueName: \"kubernetes.io/projected/03499319-d633-4824-a994-f8390fbaf20e-kube-api-access-gkqvr\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.787333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.787039    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-metrics-certs\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.787333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.787299    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03499319-d633-4824-a994-f8390fbaf20e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.788215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.788181    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03499319-d633-4824-a994-f8390fbaf20e-openshift-service-ca\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.790131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.790082    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-metrics-certs\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.790757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.790731    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-maas-api-tls\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.805977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.805951    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkqvr\" (UniqueName: \"kubernetes.io/projected/03499319-d633-4824-a994-f8390fbaf20e-kube-api-access-gkqvr\") pod \"maas-api-e2e-api-a-8fd232-69f44898d6-bh2br\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:47.989888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:47.989527    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:48.324096 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:22:48.324059    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03499319_d633_4824_a994_f8390fbaf20e.slice/crio-4e454367c70ba75e15fa89d11e80521e799950fe5768d20a48a0793edbb0d3e7 WatchSource:0}: Error finding container 4e454367c70ba75e15fa89d11e80521e799950fe5768d20a48a0793edbb0d3e7: Status 404 returned error can't find the container with id 4e454367c70ba75e15fa89d11e80521e799950fe5768d20a48a0793edbb0d3e7
Nov 03 00:22:48.324590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.324565    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"]
Nov 03 00:22:48.633324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.633291    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"]
Nov 03 00:22:48.638709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.638686    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.694020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.693973    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"]
Nov 03 00:22:48.699933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.699906    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"]
Nov 03 00:22:48.706292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.706269    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.709267 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.709191    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.709437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.709417    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42bd665-182b-4e8d-870f-3793b098957d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.709704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.709683    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.709841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.709825    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.710021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.710005    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.710153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.710137    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.710283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.710247    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckz5g\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-kube-api-access-ckz5g\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.710502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.710482    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42bd665-182b-4e8d-870f-3793b098957d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.710733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.710716    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.714485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.714461    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"]
Nov 03 00:22:48.833393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.833361    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.837270 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.833633    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.846182 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.846143    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.846449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.846430    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.846606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.846592    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.846722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.846708    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckz5g\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-kube-api-access-ckz5g\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.846861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.846843    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42bd665-182b-4e8d-870f-3793b098957d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.846972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.833772    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.847088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.846975    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.847314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.834006    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.847576 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.847536    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.847707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.847690    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.847821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.847806    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brhjx\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-kube-api-access-brhjx\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.847933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.847919    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.848040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.848027    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.848150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.848135    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.848311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.848297    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42bd665-182b-4e8d-870f-3793b098957d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.848475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.848456    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.848599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.848580    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.848755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.848738    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.848947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.848909    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.850151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.850129    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.851155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.851137    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42bd665-182b-4e8d-870f-3793b098957d-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.855355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.851914    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42bd665-182b-4e8d-870f-3793b098957d-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.855478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.854597    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.885348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.861381    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br" event={"ID":"03499319-d633-4824-a994-f8390fbaf20e","Type":"ContainerStarted","Data":"4e454367c70ba75e15fa89d11e80521e799950fe5768d20a48a0793edbb0d3e7"}
Nov 03 00:22:48.885348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.877341    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:48.932006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.931952    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.935121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.935089    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckz5g\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-kube-api-access-ckz5g\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951052    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951340    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brhjx\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-kube-api-access-brhjx\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951386    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951412    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951476    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951550    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951661    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951721    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951820    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.951912    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952576 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.952157    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.952776    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.952901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.952863    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.953279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.953248    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.955407 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.955371    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.955686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.955669    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.970243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.970218    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:48.970587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:48.970567    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brhjx\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-kube-api-access-brhjx\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:49.000221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.000201    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:49.037784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.037503    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:49.407569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.407519    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br" podStartSLOduration=2.082194553 podStartE2EDuration="2.407502032s" totalImagesPullingTime="325.307479ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:48.931957527 +0000 UTC m=+1006.487309553" watchObservedRunningTime="2026-11-03 00:22:49.407502032 +0000 UTC m=+1006.962854055"
Nov 03 00:22:49.408076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.408054    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"]
Nov 03 00:22:49.409213 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:22:49.409181    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd42bd665_182b_4e8d_870f_3793b098957d.slice/crio-9202ff350e44c5cc5baa8d9fb2009f00a85bffbc8e50e796803d27f88aecbba3 WatchSource:0}: Error finding container 9202ff350e44c5cc5baa8d9fb2009f00a85bffbc8e50e796803d27f88aecbba3: Status 404 returned error can't find the container with id 9202ff350e44c5cc5baa8d9fb2009f00a85bffbc8e50e796803d27f88aecbba3
Nov 03 00:22:49.412268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.412216    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:49.412348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.412335    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:49.412394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.412379    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:49.504868 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:22:49.504838    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d22b28f_afb0_4b8c_96a8_31e7cdb63c50.slice/crio-1b0febe950669a8226920dc3bf1dc803bf535bfa358b9190e6f27ae227965201 WatchSource:0}: Error finding container 1b0febe950669a8226920dc3bf1dc803bf535bfa358b9190e6f27ae227965201: Status 404 returned error can't find the container with id 1b0febe950669a8226920dc3bf1dc803bf535bfa358b9190e6f27ae227965201
Nov 03 00:22:49.505027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.504943    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"]
Nov 03 00:22:49.507811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.507777    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:49.507927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.507855    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:49.507927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.507900    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:22:49.870683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.870647    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" event={"ID":"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50","Type":"ContainerStarted","Data":"65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b"}
Nov 03 00:22:49.871178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.870710    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" event={"ID":"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50","Type":"ContainerStarted","Data":"1b0febe950669a8226920dc3bf1dc803bf535bfa358b9190e6f27ae227965201"}
Nov 03 00:22:49.874503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.874331    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" event={"ID":"d42bd665-182b-4e8d-870f-3793b098957d","Type":"ContainerStarted","Data":"e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb"}
Nov 03 00:22:49.874503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.874391    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" event={"ID":"d42bd665-182b-4e8d-870f-3793b098957d","Type":"ContainerStarted","Data":"9202ff350e44c5cc5baa8d9fb2009f00a85bffbc8e50e796803d27f88aecbba3"}
Nov 03 00:22:49.878131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.877308    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br" event={"ID":"03499319-d633-4824-a994-f8390fbaf20e","Type":"ContainerStarted","Data":"0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522"}
Nov 03 00:22:49.905445 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:22:49.905404    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:22:49.915028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.914972    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podStartSLOduration=1.9149576320000001 podStartE2EDuration="1.914957632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:49.912432428 +0000 UTC m=+1007.467784448" watchObservedRunningTime="2026-11-03 00:22:49.914957632 +0000 UTC m=+1007.470309662"
Nov 03 00:22:49.955568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:49.955516    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podStartSLOduration=1.955502804 podStartE2EDuration="1.955502804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:22:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:22:49.952693325 +0000 UTC m=+1007.508045349" watchObservedRunningTime="2026-11-03 00:22:49.955502804 +0000 UTC m=+1007.510854827"
Nov 03 00:22:50.001504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.001466    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:50.003494 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.003460    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 00:22:50.003661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.003521    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 00:22:50.043912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.043866    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:50.045808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.045772    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 00:22:50.045978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.045834    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 00:22:50.733638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.733605    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:22:50.737518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:50.737485    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:22:51.010338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:51.007667    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.010338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:51.007733    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 00:22:51.043605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:51.043577    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 00:22:51.043860 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:51.043827    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 00:22:52.011404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:52.011366    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 00:22:52.012050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:52.012017    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 00:22:52.042967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:52.042927    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 00:22:52.043145 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:52.042979    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 00:22:53.002673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:53.002634    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 00:22:53.002996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:53.002964    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 00:22:53.042916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:53.042880    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 00:22:53.043352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:53.042932    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 00:22:54.018279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:54.017401    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 00:22:54.018279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:54.017469    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 00:22:54.043137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:54.042897    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 00:22:54.043137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:54.042948    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 00:22:55.001876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:55.001842    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 00:22:55.002184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:55.002145    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 00:22:55.043092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:55.043063    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 00:22:55.043294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:55.043111    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 00:22:55.891074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:55.891042    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:22:56.000848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:56.000807    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body=
Nov 03 00:22:56.001051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:56.000897    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused"
Nov 03 00:22:56.061328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:56.053176    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 03 00:22:56.061328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:56.053271    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 03 00:22:57.132218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:57.132176    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:57.134134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:57.134101    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:57.135580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:57.135556    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:22:57.221806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:57.221778    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:57.222680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:57.222472    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:57.223959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:57.223937    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:22:59.960338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:22:59.950577    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:23:12.203380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.203346    6000 generic.go:356] "Generic (PLEG): container finished" podID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerID="731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475" exitCode=0
Nov 03 00:23:12.203936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.203430    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" event={"ID":"6ffad482-272b-472c-aaa8-0641a2c7d24f","Type":"ContainerDied","Data":"731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475"}
Nov 03 00:23:12.204023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.204004    6000 scope.go:122] "RemoveContainer" containerID="731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475"
Nov 03 00:23:12.206186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.206145    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:23:12.206361 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.206333    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:23:12.206454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.206432    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:23:12.683755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.683721    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"]
Nov 03 00:23:12.687150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.687132    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:12.700929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.700904    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-714512\""
Nov 03 00:23:12.701072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.701052    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-714512-dockercfg-fkrqn\""
Nov 03 00:23:12.705358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.705306    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"]
Nov 03 00:23:12.805286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.799381    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:12.805286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.799512    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwnvc\" (UniqueName: \"kubernetes.io/projected/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-kube-api-access-zwnvc\") pod \"payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:12.929321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:12.925321    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"]
Nov 03 00:23:13.078376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.042860    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:13.078376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.048356    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:13.078376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.053925    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwnvc\" (UniqueName: \"kubernetes.io/projected/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-kube-api-access-zwnvc\") pod \"payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:13.078376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.054074    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.078376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.069755    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"]
Nov 03 00:23:13.112291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.102943    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwnvc\" (UniqueName: \"kubernetes.io/projected/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-kube-api-access-zwnvc\") pod \"payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:13.155168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.155131    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxbfj\" (UniqueName: \"kubernetes.io/projected/7b29e82f-0f29-4c92-a07e-c22954e5ce37-kube-api-access-bxbfj\") pod \"payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") " pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.155436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.155414    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b29e82f-0f29-4c92-a07e-c22954e5ce37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") " pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.221077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.221039    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" event={"ID":"6ffad482-272b-472c-aaa8-0641a2c7d24f","Type":"ContainerStarted","Data":"c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5"}
Nov 03 00:23:13.257045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.257007    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxbfj\" (UniqueName: \"kubernetes.io/projected/7b29e82f-0f29-4c92-a07e-c22954e5ce37-kube-api-access-bxbfj\") pod \"payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") " pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.257264 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.257236    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b29e82f-0f29-4c92-a07e-c22954e5ce37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") " pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.257943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.257921    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b29e82f-0f29-4c92-a07e-c22954e5ce37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") " pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.275266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.275223    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxbfj\" (UniqueName: \"kubernetes.io/projected/7b29e82f-0f29-4c92-a07e-c22954e5ce37-kube-api-access-bxbfj\") pod \"payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") " pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.306585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.306546    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:13.453411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.452701    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:13.632991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.622441    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"]
Nov 03 00:23:13.632991 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:23:13.626505    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c9a2bdc_a634_4179_b9ed_3c219dd6b297.slice/crio-45b28bd754c88a6025eafa97581846c3648e74e2b09689071d6cfd7f01b4079d WatchSource:0}: Error finding container 45b28bd754c88a6025eafa97581846c3648e74e2b09689071d6cfd7f01b4079d: Status 404 returned error can't find the container with id 45b28bd754c88a6025eafa97581846c3648e74e2b09689071d6cfd7f01b4079d
Nov 03 00:23:13.783778 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:23:13.783750    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b29e82f_0f29_4c92_a07e_c22954e5ce37.slice/crio-84c4882f2fe062f72c9e12c887f077c2ee2a54e884f1eb3c3e38dd6a7d190fd2 WatchSource:0}: Error finding container 84c4882f2fe062f72c9e12c887f077c2ee2a54e884f1eb3c3e38dd6a7d190fd2: Status 404 returned error can't find the container with id 84c4882f2fe062f72c9e12c887f077c2ee2a54e884f1eb3c3e38dd6a7d190fd2
Nov 03 00:23:13.785182 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.785155    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"]
Nov 03 00:23:13.875525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.875496    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:23:13.875763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.875745    6000 patch_prober.go:29] interesting pod/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:23:13.875810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:13.875792    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 00:23:14.229313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.229193    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p" event={"ID":"7b29e82f-0f29-4c92-a07e-c22954e5ce37","Type":"ContainerStarted","Data":"26c804b03a89fd6fa4d3604c0ccc4c5da56df0093ee0e1fffbcd2efa32bff0f6"}
Nov 03 00:23:14.229313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.229248    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p" event={"ID":"7b29e82f-0f29-4c92-a07e-c22954e5ce37","Type":"ContainerStarted","Data":"84c4882f2fe062f72c9e12c887f077c2ee2a54e884f1eb3c3e38dd6a7d190fd2"}
Nov 03 00:23:14.229313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.229314    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:14.230941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.230915    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx" event={"ID":"8c9a2bdc-a634-4179-b9ed-3c219dd6b297","Type":"ContainerStarted","Data":"adad5ff269366ac53aef5622573081b67313e481a3fe03352ce90084a079c818"}
Nov 03 00:23:14.231053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.230957    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx" event={"ID":"8c9a2bdc-a634-4179-b9ed-3c219dd6b297","Type":"ContainerStarted","Data":"45b28bd754c88a6025eafa97581846c3648e74e2b09689071d6cfd7f01b4079d"}
Nov 03 00:23:14.231111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.231090    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:14.267760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.267715    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p" podStartSLOduration=2.267701476 podStartE2EDuration="2.267701476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:14.265786206 +0000 UTC m=+1031.821138250" watchObservedRunningTime="2026-11-03 00:23:14.267701476 +0000 UTC m=+1031.823053500"
Nov 03 00:23:14.321333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.321249    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx" podStartSLOduration=2.321230343 podStartE2EDuration="2.321230343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:14.318944889 +0000 UTC m=+1031.874296917" watchObservedRunningTime="2026-11-03 00:23:14.321230343 +0000 UTC m=+1031.876582367"
Nov 03 00:23:14.876512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.876472    6000 patch_prober.go:29] interesting pod/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:23:14.876698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:14.876531    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 00:23:15.878281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:15.876741    6000 patch_prober.go:29] interesting pod/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:23:15.878281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:15.876803    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 00:23:16.876236 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:16.876202    6000 patch_prober.go:29] interesting pod/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:23:16.876438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:16.876288    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 00:23:17.875589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:17.875553    6000 patch_prober.go:29] interesting pod/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 03 00:23:17.875991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:17.875606    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 03 00:23:18.881195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:18.881165    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:23:18.881629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:18.881472    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:23:18.882322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:18.882305    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:23:19.899669 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:23:19.899627    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:23:24.606807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.606773    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"]
Nov 03 00:23:24.611192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.611171    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.645422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.638345    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-714512-serving-cert\""
Nov 03 00:23:24.645422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.643464    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"]
Nov 03 00:23:24.693852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.693817    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-metrics-certs\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.694084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.694060    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-maas-api-tls\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.694267 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.694219    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-openshift-service-ca\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.694350 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.694313    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9jq4\" (UniqueName: \"kubernetes.io/projected/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-kube-api-access-p9jq4\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.812283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.796387    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-maas-api-tls\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.812283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.797305    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-openshift-service-ca\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.812283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.798508    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-openshift-service-ca\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.812283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.799388    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9jq4\" (UniqueName: \"kubernetes.io/projected/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-kube-api-access-p9jq4\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.820884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.814404    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-metrics-certs\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.823329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.823214    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-maas-api-tls\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.844289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.829343    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9jq4\" (UniqueName: \"kubernetes.io/projected/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-kube-api-access-p9jq4\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.859335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.858317    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-metrics-certs\") pod \"maas-api-e2e-api-b-714512-598d787cd8-6m855\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:24.927200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:24.926906    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:25.236929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:25.236898    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"]
Nov 03 00:23:25.237304 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:23:25.237276    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88a9e4b_5a0f_4f39_8aea_b65af2b20722.slice/crio-a334214674e44b2261569310c74f262e1f615fa727c2bfd514f6ed8681ae7176 WatchSource:0}: Error finding container a334214674e44b2261569310c74f262e1f615fa727c2bfd514f6ed8681ae7176: Status 404 returned error can't find the container with id a334214674e44b2261569310c74f262e1f615fa727c2bfd514f6ed8681ae7176
Nov 03 00:23:25.240840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:25.240821    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:23:25.240957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:25.240944    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:23:25.327044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:25.327002    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855" event={"ID":"c88a9e4b-5a0f-4f39-8aea-b65af2b20722","Type":"ContainerStarted","Data":"a334214674e44b2261569310c74f262e1f615fa727c2bfd514f6ed8681ae7176"}
Nov 03 00:23:26.336008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:26.335967    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855" event={"ID":"c88a9e4b-5a0f-4f39-8aea-b65af2b20722","Type":"ContainerStarted","Data":"d8e9bf2c59510e975be4695cb85d3124ded2363b13081b0572234c7f23267821"}
Nov 03 00:23:26.336464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:26.336032    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:26.379222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:26.379163    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855" podStartSLOduration=2.026470129 podStartE2EDuration="2.379149826s" totalImagesPullingTime="352.679697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:23:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:23:26.376461338 +0000 UTC m=+1043.931813375" watchObservedRunningTime="2026-11-03 00:23:26.379149826 +0000 UTC m=+1043.934501849"
Nov 03 00:23:29.557151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:29.557117    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"]
Nov 03 00:23:32.348781 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:32.348751    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:23:34.557591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:34.557552    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" podUID="c53373dc-5938-4bc0-946d-4b064888f165" containerName="payload-processing" containerID="cri-o://f70c4b0a97b96ec319e1793ee9bdf9be59be76b1245e557d4943b192ef1bfd4c" gracePeriod=25
Nov 03 00:23:40.796558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:40.796514    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" podUID="c53373dc-5938-4bc0-946d-4b064888f165" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:23:49.897490 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:23:49.897449    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:23:50.796960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:50.796920    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" podUID="c53373dc-5938-4bc0-946d-4b064888f165" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:23:53.328378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:53.328345    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:23:53.715679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:53.715642    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"]
Nov 03 00:23:53.747246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:53.747212    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"]
Nov 03 00:23:54.741734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.741705    6000 generic.go:356] "Generic (PLEG): container finished" podID="c53373dc-5938-4bc0-946d-4b064888f165" containerID="f70c4b0a97b96ec319e1793ee9bdf9be59be76b1245e557d4943b192ef1bfd4c" exitCode=0
Nov 03 00:23:54.742157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.741787    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" event={"ID":"c53373dc-5938-4bc0-946d-4b064888f165","Type":"ContainerDied","Data":"f70c4b0a97b96ec319e1793ee9bdf9be59be76b1245e557d4943b192ef1bfd4c"}
Nov 03 00:23:54.778887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.778855    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:23:54.864032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.864001    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c53373dc-5938-4bc0-946d-4b064888f165-kube-api-access-wzl5g\" (UniqueName: \"kubernetes.io/projected/c53373dc-5938-4bc0-946d-4b064888f165-kube-api-access-wzl5g\") pod \"c53373dc-5938-4bc0-946d-4b064888f165\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") "
Nov 03 00:23:54.864214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.864099    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c53373dc-5938-4bc0-946d-4b064888f165-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53373dc-5938-4bc0-946d-4b064888f165-plugins-config-volume\") pod \"c53373dc-5938-4bc0-946d-4b064888f165\" (UID: \"c53373dc-5938-4bc0-946d-4b064888f165\") "
Nov 03 00:23:54.864501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.864477    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c53373dc-5938-4bc0-946d-4b064888f165-plugins-config-volume" pod "c53373dc-5938-4bc0-946d-4b064888f165" (UID: "c53373dc-5938-4bc0-946d-4b064888f165"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:54.864777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.864717    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c53373dc-5938-4bc0-946d-4b064888f165-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:54.866501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.866473    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c53373dc-5938-4bc0-946d-4b064888f165-kube-api-access-wzl5g" pod "c53373dc-5938-4bc0-946d-4b064888f165" (UID: "c53373dc-5938-4bc0-946d-4b064888f165"). InnerVolumeSpecName "kube-api-access-wzl5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:54.966638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:54.966603    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzl5g\" (UniqueName: \"kubernetes.io/projected/c53373dc-5938-4bc0-946d-4b064888f165-kube-api-access-wzl5g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:55.749921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:55.749895    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"
Nov 03 00:23:55.749921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:55.749911    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b" event={"ID":"c53373dc-5938-4bc0-946d-4b064888f165","Type":"ContainerDied","Data":"7f6e66257a3fc640fe99d20117d0c15465a9c2c4a1aaecfccfd1318d8b8d985f"}
Nov 03 00:23:55.750450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:55.749977    6000 scope.go:122] "RemoveContainer" containerID="f70c4b0a97b96ec319e1793ee9bdf9be59be76b1245e557d4943b192ef1bfd4c"
Nov 03 00:23:55.800158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:55.800129    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"]
Nov 03 00:23:55.811536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:55.811511    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-hhf7b"]
Nov 03 00:23:55.996397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:55.996366    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"]
Nov 03 00:23:55.996843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:55.996808    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c" podUID="90423226-4361-4905-9ac2-f1850aaa0996" containerName="maas-api" containerID="cri-o://6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013" gracePeriod=30
Nov 03 00:23:56.450282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.440449    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:23:56.528901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.528867    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-maas-api-tls\") pod \"90423226-4361-4905-9ac2-f1850aaa0996\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") "
Nov 03 00:23:56.529273 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.529233    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90423226-4361-4905-9ac2-f1850aaa0996-kube-api-access-jcwff\" (UniqueName: \"kubernetes.io/projected/90423226-4361-4905-9ac2-f1850aaa0996-kube-api-access-jcwff\") pod \"90423226-4361-4905-9ac2-f1850aaa0996\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") "
Nov 03 00:23:56.529459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.529440    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90423226-4361-4905-9ac2-f1850aaa0996-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90423226-4361-4905-9ac2-f1850aaa0996-openshift-service-ca\") pod \"90423226-4361-4905-9ac2-f1850aaa0996\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") "
Nov 03 00:23:56.529611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.529593    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-metrics-certs\") pod \"90423226-4361-4905-9ac2-f1850aaa0996\" (UID: \"90423226-4361-4905-9ac2-f1850aaa0996\") "
Nov 03 00:23:56.533782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.532002    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90423226-4361-4905-9ac2-f1850aaa0996-openshift-service-ca" pod "90423226-4361-4905-9ac2-f1850aaa0996" (UID: "90423226-4361-4905-9ac2-f1850aaa0996"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:56.562282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.548433    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-metrics-certs" pod "90423226-4361-4905-9ac2-f1850aaa0996" (UID: "90423226-4361-4905-9ac2-f1850aaa0996"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:23:56.578229 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.577407    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-maas-api-tls" pod "90423226-4361-4905-9ac2-f1850aaa0996" (UID: "90423226-4361-4905-9ac2-f1850aaa0996"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:23:56.585942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.585796    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90423226-4361-4905-9ac2-f1850aaa0996-kube-api-access-jcwff" pod "90423226-4361-4905-9ac2-f1850aaa0996" (UID: "90423226-4361-4905-9ac2-f1850aaa0996"). InnerVolumeSpecName "kube-api-access-jcwff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:56.668420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.667485    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:56.668420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.667510    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcwff\" (UniqueName: \"kubernetes.io/projected/90423226-4361-4905-9ac2-f1850aaa0996-kube-api-access-jcwff\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:56.668420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.667519    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90423226-4361-4905-9ac2-f1850aaa0996-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:56.668420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.667527    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90423226-4361-4905-9ac2-f1850aaa0996-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:56.699299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.671908    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c53373dc-5938-4bc0-946d-4b064888f165" path="/var/lib/kubelet/pods/c53373dc-5938-4bc0-946d-4b064888f165/volumes"
Nov 03 00:23:56.763754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.763719    6000 generic.go:356] "Generic (PLEG): container finished" podID="90423226-4361-4905-9ac2-f1850aaa0996" containerID="6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013" exitCode=0
Nov 03 00:23:56.763754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.763760    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c" event={"ID":"90423226-4361-4905-9ac2-f1850aaa0996","Type":"ContainerDied","Data":"6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013"}
Nov 03 00:23:56.764337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.763775    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"
Nov 03 00:23:56.764337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.763807    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c" event={"ID":"90423226-4361-4905-9ac2-f1850aaa0996","Type":"ContainerDied","Data":"a81292e809d282fa31cb4a613fbafdf424ce914c8b68f9b3b639d9c4e92e08b8"}
Nov 03 00:23:56.764337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.763831    6000 scope.go:122] "RemoveContainer" containerID="6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013"
Nov 03 00:23:56.778683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.778660    6000 scope.go:122] "RemoveContainer" containerID="6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013"
Nov 03 00:23:56.779159 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:23:56.779121    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013\": container with ID starting with 6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013 not found: ID does not exist" containerID="6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013"
Nov 03 00:23:56.779312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.779163    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013"} err="failed to get container status \"6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013\": rpc error: code = NotFound desc = could not find container \"6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013\": container with ID starting with 6188d3dea097132e021c96dd5db43a6bbbc8759d1351f18361dcf7cb718b0013 not found: ID does not exist"
Nov 03 00:23:56.812728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.812698    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"]
Nov 03 00:23:56.816286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:56.816250    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-m468c"]
Nov 03 00:23:58.567908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:58.567873    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90423226-4361-4905-9ac2-f1850aaa0996" path="/var/lib/kubelet/pods/90423226-4361-4905-9ac2-f1850aaa0996/volumes"
Nov 03 00:23:58.716525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:58.716481    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk" podUID="c8ee1980-21d0-4d91-bc30-eae642827307" containerName="payload-processing" containerID="cri-o://e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813" gracePeriod=25
Nov 03 00:23:58.748563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:58.748530    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn" podUID="45ce9835-2515-423a-adeb-04f07c324e63" containerName="payload-pre-processing" containerID="cri-o://fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43" gracePeriod=25
Nov 03 00:23:59.043544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.043510    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:23:59.068529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.068507    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:23:59.125338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.123796    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45ce9835-2515-423a-adeb-04f07c324e63-kube-api-access-twmbp\" (UniqueName: \"kubernetes.io/projected/45ce9835-2515-423a-adeb-04f07c324e63-kube-api-access-twmbp\") pod \"45ce9835-2515-423a-adeb-04f07c324e63\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") "
Nov 03 00:23:59.125338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.123865    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8ee1980-21d0-4d91-bc30-eae642827307-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee1980-21d0-4d91-bc30-eae642827307-plugins-config-volume\") pod \"c8ee1980-21d0-4d91-bc30-eae642827307\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") "
Nov 03 00:23:59.125338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.124041    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8ee1980-21d0-4d91-bc30-eae642827307-kube-api-access-gtczm\" (UniqueName: \"kubernetes.io/projected/c8ee1980-21d0-4d91-bc30-eae642827307-kube-api-access-gtczm\") pod \"c8ee1980-21d0-4d91-bc30-eae642827307\" (UID: \"c8ee1980-21d0-4d91-bc30-eae642827307\") "
Nov 03 00:23:59.125338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.124225    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45ce9835-2515-423a-adeb-04f07c324e63-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ce9835-2515-423a-adeb-04f07c324e63-plugins-config-volume\") pod \"45ce9835-2515-423a-adeb-04f07c324e63\" (UID: \"45ce9835-2515-423a-adeb-04f07c324e63\") "
Nov 03 00:23:59.160283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.154491    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8ee1980-21d0-4d91-bc30-eae642827307-plugins-config-volume" pod "c8ee1980-21d0-4d91-bc30-eae642827307" (UID: "c8ee1980-21d0-4d91-bc30-eae642827307"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:59.191671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.184334    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45ce9835-2515-423a-adeb-04f07c324e63-plugins-config-volume" pod "45ce9835-2515-423a-adeb-04f07c324e63" (UID: "45ce9835-2515-423a-adeb-04f07c324e63"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:23:59.191671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.186822    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45ce9835-2515-423a-adeb-04f07c324e63-kube-api-access-twmbp" pod "45ce9835-2515-423a-adeb-04f07c324e63" (UID: "45ce9835-2515-423a-adeb-04f07c324e63"). InnerVolumeSpecName "kube-api-access-twmbp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:59.191671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.186956    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8ee1980-21d0-4d91-bc30-eae642827307-kube-api-access-gtczm" pod "c8ee1980-21d0-4d91-bc30-eae642827307" (UID: "c8ee1980-21d0-4d91-bc30-eae642827307"). InnerVolumeSpecName "kube-api-access-gtczm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:23:59.239718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.239690    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gtczm\" (UniqueName: \"kubernetes.io/projected/c8ee1980-21d0-4d91-bc30-eae642827307-kube-api-access-gtczm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:59.239718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.239716    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45ce9835-2515-423a-adeb-04f07c324e63-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:59.239925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.239730    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twmbp\" (UniqueName: \"kubernetes.io/projected/45ce9835-2515-423a-adeb-04f07c324e63-kube-api-access-twmbp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:59.239925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.239738    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c8ee1980-21d0-4d91-bc30-eae642827307-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:23:59.791057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.791017    6000 generic.go:356] "Generic (PLEG): container finished" podID="45ce9835-2515-423a-adeb-04f07c324e63" containerID="fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43" exitCode=0
Nov 03 00:23:59.791626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.791084    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"
Nov 03 00:23:59.791626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.791100    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn" event={"ID":"45ce9835-2515-423a-adeb-04f07c324e63","Type":"ContainerDied","Data":"fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43"}
Nov 03 00:23:59.791626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.791210    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn" event={"ID":"45ce9835-2515-423a-adeb-04f07c324e63","Type":"ContainerDied","Data":"117de34c98118d87d67fa3faf1d48f67a95b31454cec0fcc710ad6212bc00e02"}
Nov 03 00:23:59.791626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.791247    6000 scope.go:122] "RemoveContainer" containerID="fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43"
Nov 03 00:23:59.793740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.793713    6000 generic.go:356] "Generic (PLEG): container finished" podID="c8ee1980-21d0-4d91-bc30-eae642827307" containerID="e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813" exitCode=0
Nov 03 00:23:59.793868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.793793    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk" event={"ID":"c8ee1980-21d0-4d91-bc30-eae642827307","Type":"ContainerDied","Data":"e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813"}
Nov 03 00:23:59.793868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.793808    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"
Nov 03 00:23:59.793868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.793834    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk" event={"ID":"c8ee1980-21d0-4d91-bc30-eae642827307","Type":"ContainerDied","Data":"6a0d6f9bda54f31861f14c88c22a2d3cfe3af70766dc4167c63c673775732a09"}
Nov 03 00:23:59.804553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.804538    6000 scope.go:122] "RemoveContainer" containerID="fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43"
Nov 03 00:23:59.805616 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:23:59.805590    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43\": container with ID starting with fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43 not found: ID does not exist" containerID="fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43"
Nov 03 00:23:59.805720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.805627    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43"} err="failed to get container status \"fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43\": rpc error: code = NotFound desc = could not find container \"fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43\": container with ID starting with fdbb219e6dc1af5ff8ca7b1897cb6519955216f405547aa5d23046d19283fc43 not found: ID does not exist"
Nov 03 00:23:59.805720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.805651    6000 scope.go:122] "RemoveContainer" containerID="e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813"
Nov 03 00:23:59.815290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.815249    6000 scope.go:122] "RemoveContainer" containerID="e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813"
Nov 03 00:23:59.815577 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:23:59.815556    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813\": container with ID starting with e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813 not found: ID does not exist" containerID="e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813"
Nov 03 00:23:59.815652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.815584    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813"} err="failed to get container status \"e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813\": rpc error: code = NotFound desc = could not find container \"e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813\": container with ID starting with e1313c7c2fad58033d5ece7e5f6d749507b648afc443200f906d5711d45d2813 not found: ID does not exist"
Nov 03 00:23:59.840285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.840241    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"]
Nov 03 00:23:59.854649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.854622    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-8fd232-6b9789dc4c-zdknn"]
Nov 03 00:23:59.880395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.880364    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"]
Nov 03 00:23:59.884984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:23:59.884962    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-8fd232-fd8d9c969-l25qk"]
Nov 03 00:24:00.568674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.568639    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45ce9835-2515-423a-adeb-04f07c324e63" path="/var/lib/kubelet/pods/45ce9835-2515-423a-adeb-04f07c324e63/volumes"
Nov 03 00:24:00.569076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.569063    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8ee1980-21d0-4d91-bc30-eae642827307" path="/var/lib/kubelet/pods/c8ee1980-21d0-4d91-bc30-eae642827307/volumes"
Nov 03 00:24:00.597650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.597611    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"]
Nov 03 00:24:00.598388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598370    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c8ee1980-21d0-4d91-bc30-eae642827307" containerName="payload-processing"
Nov 03 00:24:00.598449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598389    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8ee1980-21d0-4d91-bc30-eae642827307" containerName="payload-processing"
Nov 03 00:24:00.598449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598411    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c53373dc-5938-4bc0-946d-4b064888f165" containerName="payload-processing"
Nov 03 00:24:00.598449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598417    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c53373dc-5938-4bc0-946d-4b064888f165" containerName="payload-processing"
Nov 03 00:24:00.598449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598425    6000 cpu_manager.go:414] "containerMap: removing container" podUID="90423226-4361-4905-9ac2-f1850aaa0996" containerName="maas-api"
Nov 03 00:24:00.598449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598429    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90423226-4361-4905-9ac2-f1850aaa0996" containerName="maas-api"
Nov 03 00:24:00.598449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598442    6000 cpu_manager.go:414] "containerMap: removing container" podUID="45ce9835-2515-423a-adeb-04f07c324e63" containerName="payload-pre-processing"
Nov 03 00:24:00.598449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598447    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45ce9835-2515-423a-adeb-04f07c324e63" containerName="payload-pre-processing"
Nov 03 00:24:00.598663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598592    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c53373dc-5938-4bc0-946d-4b064888f165" containerName="payload-processing"
Nov 03 00:24:00.598663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598602    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="90423226-4361-4905-9ac2-f1850aaa0996" containerName="maas-api"
Nov 03 00:24:00.598663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598610    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="45ce9835-2515-423a-adeb-04f07c324e63" containerName="payload-pre-processing"
Nov 03 00:24:00.598663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.598624    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8ee1980-21d0-4d91-bc30-eae642827307" containerName="payload-processing"
Nov 03 00:24:00.604087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.604069    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.621170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.621150    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"]
Nov 03 00:24:00.631297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.629596    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-5c5686-serving-cert\""
Nov 03 00:24:00.669181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.660020    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-maas-api-tls\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.669181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.660321    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-metrics-certs\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.669181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.660566    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec23fd-b176-4386-a861-277c2dde2591-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.669181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.660716    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xdzv\" (UniqueName: \"kubernetes.io/projected/0fec23fd-b176-4386-a861-277c2dde2591-kube-api-access-6xdzv\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.762093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.762050    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-maas-api-tls\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.762343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.762165    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-metrics-certs\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.762343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.762212    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec23fd-b176-4386-a861-277c2dde2591-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.762343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.762244    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xdzv\" (UniqueName: \"kubernetes.io/projected/0fec23fd-b176-4386-a861-277c2dde2591-kube-api-access-6xdzv\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.762938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.762908    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec23fd-b176-4386-a861-277c2dde2591-openshift-service-ca\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.764993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.764968    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-maas-api-tls\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.765110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.764968    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-metrics-certs\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.780463 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.780428    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xdzv\" (UniqueName: \"kubernetes.io/projected/0fec23fd-b176-4386-a861-277c2dde2591-kube-api-access-6xdzv\") pod \"maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:00.935282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:00.933909    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:01.287199 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:01.287172    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fec23fd_b176_4386_a861_277c2dde2591.slice/crio-88e1fb95ea71477ef9a480b714390bc846e9a3892eb6944fc2ca9c98313510a7 WatchSource:0}: Error finding container 88e1fb95ea71477ef9a480b714390bc846e9a3892eb6944fc2ca9c98313510a7: Status 404 returned error can't find the container with id 88e1fb95ea71477ef9a480b714390bc846e9a3892eb6944fc2ca9c98313510a7
Nov 03 00:24:01.287662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:01.287638    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"]
Nov 03 00:24:01.811294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:01.811244    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2" event={"ID":"0fec23fd-b176-4386-a861-277c2dde2591","Type":"ContainerStarted","Data":"dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5"}
Nov 03 00:24:01.811450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:01.811329    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2" event={"ID":"0fec23fd-b176-4386-a861-277c2dde2591","Type":"ContainerStarted","Data":"88e1fb95ea71477ef9a480b714390bc846e9a3892eb6944fc2ca9c98313510a7"}
Nov 03 00:24:01.811493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:01.811458    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:01.946281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:01.899052    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2" podStartSLOduration=1.4817621029999999 podStartE2EDuration="1.899029164s" totalImagesPullingTime="417.267061ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:01.861785379 +0000 UTC m=+1079.417137406" watchObservedRunningTime="2026-11-03 00:24:01.899029164 +0000 UTC m=+1079.454381189"
Nov 03 00:24:04.123920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.123886    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"]
Nov 03 00:24:04.124365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.124077    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br" podUID="03499319-d633-4824-a994-f8390fbaf20e" containerName="maas-api" containerID="cri-o://0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522" gracePeriod=30
Nov 03 00:24:04.529877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.529839    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:24:04.682778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.682694    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-metrics-certs\") pod \"03499319-d633-4824-a994-f8390fbaf20e\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") "
Nov 03 00:24:04.682949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.682812    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-maas-api-tls\") pod \"03499319-d633-4824-a994-f8390fbaf20e\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") "
Nov 03 00:24:04.682949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.682942    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03499319-d633-4824-a994-f8390fbaf20e-kube-api-access-gkqvr\" (UniqueName: \"kubernetes.io/projected/03499319-d633-4824-a994-f8390fbaf20e-kube-api-access-gkqvr\") pod \"03499319-d633-4824-a994-f8390fbaf20e\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") "
Nov 03 00:24:04.683076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.682989    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03499319-d633-4824-a994-f8390fbaf20e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03499319-d633-4824-a994-f8390fbaf20e-openshift-service-ca\") pod \"03499319-d633-4824-a994-f8390fbaf20e\" (UID: \"03499319-d633-4824-a994-f8390fbaf20e\") "
Nov 03 00:24:04.731444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.731387    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03499319-d633-4824-a994-f8390fbaf20e-kube-api-access-gkqvr" pod "03499319-d633-4824-a994-f8390fbaf20e" (UID: "03499319-d633-4824-a994-f8390fbaf20e"). InnerVolumeSpecName "kube-api-access-gkqvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:04.745410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.737234    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03499319-d633-4824-a994-f8390fbaf20e-openshift-service-ca" pod "03499319-d633-4824-a994-f8390fbaf20e" (UID: "03499319-d633-4824-a994-f8390fbaf20e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:24:04.782599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.782548    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-metrics-certs" pod "03499319-d633-4824-a994-f8390fbaf20e" (UID: "03499319-d633-4824-a994-f8390fbaf20e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:24:04.799857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.799822    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:04.808411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.808379    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkqvr\" (UniqueName: \"kubernetes.io/projected/03499319-d633-4824-a994-f8390fbaf20e-kube-api-access-gkqvr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:04.808636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.808622    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/03499319-d633-4824-a994-f8390fbaf20e-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:04.870427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.868693    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-maas-api-tls" pod "03499319-d633-4824-a994-f8390fbaf20e" (UID: "03499319-d633-4824-a994-f8390fbaf20e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:24:04.892046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.892010    6000 generic.go:356] "Generic (PLEG): container finished" podID="03499319-d633-4824-a994-f8390fbaf20e" containerID="0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522" exitCode=0
Nov 03 00:24:04.892346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.892327    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"
Nov 03 00:24:04.893480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.893437    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br" event={"ID":"03499319-d633-4824-a994-f8390fbaf20e","Type":"ContainerDied","Data":"0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522"}
Nov 03 00:24:04.893590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.893503    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br" event={"ID":"03499319-d633-4824-a994-f8390fbaf20e","Type":"ContainerDied","Data":"4e454367c70ba75e15fa89d11e80521e799950fe5768d20a48a0793edbb0d3e7"}
Nov 03 00:24:04.893590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.893537    6000 scope.go:122] "RemoveContainer" containerID="0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522"
Nov 03 00:24:04.910998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.910965    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/03499319-d633-4824-a994-f8390fbaf20e-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:04.917588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.917566    6000 scope.go:122] "RemoveContainer" containerID="0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522"
Nov 03 00:24:04.917920 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:04.917893    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522\": container with ID starting with 0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522 not found: ID does not exist" containerID="0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522"
Nov 03 00:24:04.917969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.917935    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522"} err="failed to get container status \"0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522\": rpc error: code = NotFound desc = could not find container \"0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522\": container with ID starting with 0fc7e6180aef594317b6fb8486b3c1334a8360c5ada8220d226aaf61ff59a522 not found: ID does not exist"
Nov 03 00:24:04.953711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.953632    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"]
Nov 03 00:24:04.964896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:04.964861    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-8fd232-69f44898d6-bh2br"]
Nov 03 00:24:05.593161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.593126    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"]
Nov 03 00:24:05.593975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.593951    6000 cpu_manager.go:414] "containerMap: removing container" podUID="03499319-d633-4824-a994-f8390fbaf20e" containerName="maas-api"
Nov 03 00:24:05.593975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.593970    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03499319-d633-4824-a994-f8390fbaf20e" containerName="maas-api"
Nov 03 00:24:05.594202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.594186    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="03499319-d633-4824-a994-f8390fbaf20e" containerName="maas-api"
Nov 03 00:24:05.599227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.599209    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.613236 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.613152    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9f76b708\""
Nov 03 00:24:05.613868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.613848    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9f76b708-dockercfg-wx6gp\""
Nov 03 00:24:05.616744 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.616719    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"]
Nov 03 00:24:05.625031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.625001    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1a58a2-1ce7-4c6f-930f-b084ef702624-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.625163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.625046    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zjw2\" (UniqueName: \"kubernetes.io/projected/3d1a58a2-1ce7-4c6f-930f-b084ef702624-kube-api-access-4zjw2\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.732282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.726607    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"]
Nov 03 00:24:05.803026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.785601    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1a58a2-1ce7-4c6f-930f-b084ef702624-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.803026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.785658    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zjw2\" (UniqueName: \"kubernetes.io/projected/3d1a58a2-1ce7-4c6f-930f-b084ef702624-kube-api-access-4zjw2\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.803026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.786967    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1a58a2-1ce7-4c6f-930f-b084ef702624-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.848011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.847203    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:05.864788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.859069    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zjw2\" (UniqueName: \"kubernetes.io/projected/3d1a58a2-1ce7-4c6f-930f-b084ef702624-kube-api-access-4zjw2\") pod \"payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.887488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.887457    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1abd54d8-38bb-405a-8ed8-31b556eb33ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:05.887738 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.887721    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxxlk\" (UniqueName: \"kubernetes.io/projected/1abd54d8-38bb-405a-8ed8-31b556eb33ae-kube-api-access-fxxlk\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:05.927623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.909893    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"]
Nov 03 00:24:05.927623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.914163    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:05.989927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.989886    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxxlk\" (UniqueName: \"kubernetes.io/projected/1abd54d8-38bb-405a-8ed8-31b556eb33ae-kube-api-access-fxxlk\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:05.990221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.990200    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1abd54d8-38bb-405a-8ed8-31b556eb33ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:05.991346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:05.991315    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1abd54d8-38bb-405a-8ed8-31b556eb33ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:06.013862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.013836    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxxlk\" (UniqueName: \"kubernetes.io/projected/1abd54d8-38bb-405a-8ed8-31b556eb33ae-kube-api-access-fxxlk\") pod \"payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") " pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:06.240970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.240939    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:06.256052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.256022    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"]
Nov 03 00:24:06.256762 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:06.256736    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1a58a2_1ce7_4c6f_930f_b084ef702624.slice/crio-6823f48dd8a6f83839443be5f946915d7c04978ffea68ac20df2e24095ed562a WatchSource:0}: Error finding container 6823f48dd8a6f83839443be5f946915d7c04978ffea68ac20df2e24095ed562a: Status 404 returned error can't find the container with id 6823f48dd8a6f83839443be5f946915d7c04978ffea68ac20df2e24095ed562a
Nov 03 00:24:06.549044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.549015    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"]
Nov 03 00:24:06.549688 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:06.549660    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1abd54d8_38bb_405a_8ed8_31b556eb33ae.slice/crio-174b1f55057bdd22b6169b9115d445d8f3ddc2cf1252dc8b04716fdae7dbb7f5 WatchSource:0}: Error finding container 174b1f55057bdd22b6169b9115d445d8f3ddc2cf1252dc8b04716fdae7dbb7f5: Status 404 returned error can't find the container with id 174b1f55057bdd22b6169b9115d445d8f3ddc2cf1252dc8b04716fdae7dbb7f5
Nov 03 00:24:06.568000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.567978    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03499319-d633-4824-a994-f8390fbaf20e" path="/var/lib/kubelet/pods/03499319-d633-4824-a994-f8390fbaf20e/volumes"
Nov 03 00:24:06.961965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.961927    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj" event={"ID":"3d1a58a2-1ce7-4c6f-930f-b084ef702624","Type":"ContainerStarted","Data":"2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9"}
Nov 03 00:24:06.962432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.961991    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj" event={"ID":"3d1a58a2-1ce7-4c6f-930f-b084ef702624","Type":"ContainerStarted","Data":"6823f48dd8a6f83839443be5f946915d7c04978ffea68ac20df2e24095ed562a"}
Nov 03 00:24:06.962432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.962024    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:06.963719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.963689    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7" event={"ID":"1abd54d8-38bb-405a-8ed8-31b556eb33ae","Type":"ContainerStarted","Data":"d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a"}
Nov 03 00:24:06.963850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.963734    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7" event={"ID":"1abd54d8-38bb-405a-8ed8-31b556eb33ae","Type":"ContainerStarted","Data":"174b1f55057bdd22b6169b9115d445d8f3ddc2cf1252dc8b04716fdae7dbb7f5"}
Nov 03 00:24:06.963850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:06.963755    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:07.003647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:07.003600    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj" podStartSLOduration=2.003588099 podStartE2EDuration="2.003588099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:07.000525771 +0000 UTC m=+1084.555877827" watchObservedRunningTime="2026-11-03 00:24:07.003588099 +0000 UTC m=+1084.558940126"
Nov 03 00:24:07.039954 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:07.039905    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7" podStartSLOduration=2.03988513 podStartE2EDuration="2.03988513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:07.039042549 +0000 UTC m=+1084.594394573" watchObservedRunningTime="2026-11-03 00:24:07.03988513 +0000 UTC m=+1084.595237155"
Nov 03 00:24:07.824061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:07.824034    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:24:08.266484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:08.266445    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"]
Nov 03 00:24:08.266933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:08.266733    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy" containerID="cri-o://c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5" gracePeriod=30
Nov 03 00:24:12.347563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.347470    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"]
Nov 03 00:24:12.351890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.351866    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.367680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.367658    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-9f76b708-serving-cert\""
Nov 03 00:24:12.385334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.385296    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"]
Nov 03 00:24:12.414915 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.414880    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-metrics-certs\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.415070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.415018    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cs6j\" (UniqueName: \"kubernetes.io/projected/6ac20e9e-db09-46ae-9721-077f93df6b9c-kube-api-access-7cs6j\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.415179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.415154    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-maas-api-tls\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.415305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.415288    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac20e9e-db09-46ae-9721-077f93df6b9c-openshift-service-ca\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.516762 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.516727    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cs6j\" (UniqueName: \"kubernetes.io/projected/6ac20e9e-db09-46ae-9721-077f93df6b9c-kube-api-access-7cs6j\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.516932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.516810    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-maas-api-tls\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.516932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.516868    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac20e9e-db09-46ae-9721-077f93df6b9c-openshift-service-ca\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.517060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.517041    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-metrics-certs\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.517615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.517577    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac20e9e-db09-46ae-9721-077f93df6b9c-openshift-service-ca\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.519594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.519573    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-maas-api-tls\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.519704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.519672    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-metrics-certs\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.556938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.556901    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cs6j\" (UniqueName: \"kubernetes.io/projected/6ac20e9e-db09-46ae-9721-077f93df6b9c-kube-api-access-7cs6j\") pod \"maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:12.669546 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:12.669510    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:13.025352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:13.025323    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"]
Nov 03 00:24:13.025892 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:13.025864    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ac20e9e_db09_46ae_9721_077f93df6b9c.slice/crio-7cb5153194545a43319f59a7695aef71092b05ad9b0869985481069f68369a37 WatchSource:0}: Error finding container 7cb5153194545a43319f59a7695aef71092b05ad9b0869985481069f68369a37: Status 404 returned error can't find the container with id 7cb5153194545a43319f59a7695aef71092b05ad9b0869985481069f68369a37
Nov 03 00:24:13.963733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:13.963707    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:24:14.047555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.047440    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6" event={"ID":"6ac20e9e-db09-46ae-9721-077f93df6b9c","Type":"ContainerStarted","Data":"e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22"}
Nov 03 00:24:14.047555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.047506    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6" event={"ID":"6ac20e9e-db09-46ae-9721-077f93df6b9c","Type":"ContainerStarted","Data":"7cb5153194545a43319f59a7695aef71092b05ad9b0869985481069f68369a37"}
Nov 03 00:24:14.048635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.048608    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:14.051418 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.051388    6000 generic.go:356] "Generic (PLEG): container finished" podID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerID="c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5" exitCode=0
Nov 03 00:24:14.051547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.051438    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" event={"ID":"6ffad482-272b-472c-aaa8-0641a2c7d24f","Type":"ContainerDied","Data":"c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5"}
Nov 03 00:24:14.051547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.051479    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh" event={"ID":"6ffad482-272b-472c-aaa8-0641a2c7d24f","Type":"ContainerDied","Data":"12600acc8f668dfa590ee9b86d66a91c6fd25516f1c1c77f22763294c1f60ff9"}
Nov 03 00:24:14.051547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.051509    6000 scope.go:122] "RemoveContainer" containerID="c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5"
Nov 03 00:24:14.051709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.051672    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"
Nov 03 00:24:14.057549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.057524    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-certs\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.057742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.057649    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-kube-api-access-qdj6m\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-kube-api-access-qdj6m\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.057742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.057706    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-token\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-token\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.057878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.057765    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-envoy\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.057878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.057807    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-credential-socket\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.057998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.057899    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-socket\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.057998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.057979    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-podinfo\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.058155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.058116    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-data\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.058227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.058166    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ffad482-272b-472c-aaa8-0641a2c7d24f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffad482-272b-472c-aaa8-0641a2c7d24f-istiod-ca-cert\") pod \"6ffad482-272b-472c-aaa8-0641a2c7d24f\" (UID: \"6ffad482-272b-472c-aaa8-0641a2c7d24f\") "
Nov 03 00:24:14.059229 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.059141    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ffad482-272b-472c-aaa8-0641a2c7d24f-istiod-ca-cert" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:24:14.059494 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.059468    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-certs" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:14.067306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.062451    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-kube-api-access-qdj6m" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "kube-api-access-qdj6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:14.067306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.065584    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-token" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:14.067965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.067913    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-envoy" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:14.068141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.068121    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-credential-socket" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:14.068330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.068309    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-socket" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:14.069156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.069140    6000 scope.go:122] "RemoveContainer" containerID="731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475"
Nov 03 00:24:14.069481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.069435    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-data" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:24:14.070541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.070513    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-podinfo" pod "6ffad482-272b-472c-aaa8-0641a2c7d24f" (UID: "6ffad482-272b-472c-aaa8-0641a2c7d24f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:24:14.097834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.097786    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6" podStartSLOduration=1.7318840130000002 podStartE2EDuration="2.097769303s" totalImagesPullingTime="365.88529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:14.093387754 +0000 UTC m=+1091.648739791" watchObservedRunningTime="2026-11-03 00:24:14.097769303 +0000 UTC m=+1091.653121356"
Nov 03 00:24:14.098587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.098561    6000 scope.go:122] "RemoveContainer" containerID="c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5"
Nov 03 00:24:14.098981 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:14.098956    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5\": container with ID starting with c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5 not found: ID does not exist" containerID="c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5"
Nov 03 00:24:14.099046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.098995    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5"} err="failed to get container status \"c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5\": rpc error: code = NotFound desc = could not find container \"c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5\": container with ID starting with c75f71d81e860a516806c8224ad80a114e04e6575dc44b9293ebfd5aff541ef5 not found: ID does not exist"
Nov 03 00:24:14.099046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.099021    6000 scope.go:122] "RemoveContainer" containerID="731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475"
Nov 03 00:24:14.099397 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:14.099377    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475\": container with ID starting with 731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475 not found: ID does not exist" containerID="731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475"
Nov 03 00:24:14.099474 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.099406    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475"} err="failed to get container status \"731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475\": rpc error: code = NotFound desc = could not find container \"731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475\": container with ID starting with 731594a968b5bbff31f93b1dccfbe5a6e938b2c31a4864d85191bd320fddd475 not found: ID does not exist"
Nov 03 00:24:14.160136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160100    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160134    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6ffad482-272b-472c-aaa8-0641a2c7d24f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160148    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160160    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdj6m\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-kube-api-access-qdj6m\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160173    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160185    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160196    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160211    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6ffad482-272b-472c-aaa8-0641a2c7d24f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.160381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.160224    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6ffad482-272b-472c-aaa8-0641a2c7d24f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:14.399631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.399600    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"]
Nov 03 00:24:14.409456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.409428    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-8fd232-openshift-default-85b6db77c9-tcxjh"]
Nov 03 00:24:14.603279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.597856    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" path="/var/lib/kubelet/pods/6ffad482-272b-472c-aaa8-0641a2c7d24f/volumes"
Nov 03 00:24:14.800692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.800597    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"]
Nov 03 00:24:14.828489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:14.828459    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"]
Nov 03 00:24:17.974167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:17.974136    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:24:17.974648 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:17.974203    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:24:19.804424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:19.804364    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p" podUID="7b29e82f-0f29-4c92-a07e-c22954e5ce37" containerName="payload-processing" containerID="cri-o://26c804b03a89fd6fa4d3604c0ccc4c5da56df0093ee0e1fffbcd2efa32bff0f6" gracePeriod=25
Nov 03 00:24:19.829561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:19.829348    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx" podUID="8c9a2bdc-a634-4179-b9ed-3c219dd6b297" containerName="payload-pre-processing" containerID="cri-o://adad5ff269366ac53aef5622573081b67313e481a3fe03352ce90084a079c818" gracePeriod=25
Nov 03 00:24:20.016709 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:20.016359    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:24:20.621758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.621421    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:24:20.629936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.629572    6000 generic.go:356] "Generic (PLEG): container finished" podID="7b29e82f-0f29-4c92-a07e-c22954e5ce37" containerID="26c804b03a89fd6fa4d3604c0ccc4c5da56df0093ee0e1fffbcd2efa32bff0f6" exitCode=0
Nov 03 00:24:20.629936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.629763    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"
Nov 03 00:24:20.632578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.632405    6000 generic.go:356] "Generic (PLEG): container finished" podID="8c9a2bdc-a634-4179-b9ed-3c219dd6b297" containerID="adad5ff269366ac53aef5622573081b67313e481a3fe03352ce90084a079c818" exitCode=0
Nov 03 00:24:20.788005 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.774704    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b29e82f-0f29-4c92-a07e-c22954e5ce37-kube-api-access-bxbfj\" (UniqueName: \"kubernetes.io/projected/7b29e82f-0f29-4c92-a07e-c22954e5ce37-kube-api-access-bxbfj\") pod \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") "
Nov 03 00:24:20.788005 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.774790    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b29e82f-0f29-4c92-a07e-c22954e5ce37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b29e82f-0f29-4c92-a07e-c22954e5ce37-plugins-config-volume\") pod \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\" (UID: \"7b29e82f-0f29-4c92-a07e-c22954e5ce37\") "
Nov 03 00:24:20.788005 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.775912    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b29e82f-0f29-4c92-a07e-c22954e5ce37-plugins-config-volume" pod "7b29e82f-0f29-4c92-a07e-c22954e5ce37" (UID: "7b29e82f-0f29-4c92-a07e-c22954e5ce37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:24:20.809001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.808955    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b29e82f-0f29-4c92-a07e-c22954e5ce37-kube-api-access-bxbfj" pod "7b29e82f-0f29-4c92-a07e-c22954e5ce37" (UID: "7b29e82f-0f29-4c92-a07e-c22954e5ce37"). InnerVolumeSpecName "kube-api-access-bxbfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:20.861287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.858240    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p" event={"ID":"7b29e82f-0f29-4c92-a07e-c22954e5ce37","Type":"ContainerDied","Data":"26c804b03a89fd6fa4d3604c0ccc4c5da56df0093ee0e1fffbcd2efa32bff0f6"}
Nov 03 00:24:20.861287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.858411    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx" event={"ID":"8c9a2bdc-a634-4179-b9ed-3c219dd6b297","Type":"ContainerDied","Data":"adad5ff269366ac53aef5622573081b67313e481a3fe03352ce90084a079c818"}
Nov 03 00:24:20.861287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.858469    6000 scope.go:122] "RemoveContainer" containerID="26c804b03a89fd6fa4d3604c0ccc4c5da56df0093ee0e1fffbcd2efa32bff0f6"
Nov 03 00:24:20.882668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.882630    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxbfj\" (UniqueName: \"kubernetes.io/projected/7b29e82f-0f29-4c92-a07e-c22954e5ce37-kube-api-access-bxbfj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:20.882924 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.882891    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b29e82f-0f29-4c92-a07e-c22954e5ce37-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:20.949936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:20.949904    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:24:21.100348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.094876    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-plugins-config-volume\") pod \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") "
Nov 03 00:24:21.130573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.130541    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-kube-api-access-zwnvc\" (UniqueName: \"kubernetes.io/projected/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-kube-api-access-zwnvc\") pod \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\" (UID: \"8c9a2bdc-a634-4179-b9ed-3c219dd6b297\") "
Nov 03 00:24:21.139076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.106701    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-plugins-config-volume" pod "8c9a2bdc-a634-4179-b9ed-3c219dd6b297" (UID: "8c9a2bdc-a634-4179-b9ed-3c219dd6b297"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:24:21.176607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.176582    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:24:21.177471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.177429    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"]
Nov 03 00:24:21.179690 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.179646    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-kube-api-access-zwnvc" pod "8c9a2bdc-a634-4179-b9ed-3c219dd6b297" (UID: "8c9a2bdc-a634-4179-b9ed-3c219dd6b297"). InnerVolumeSpecName "kube-api-access-zwnvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:21.180553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.180526    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-714512-b95cf9d6f-92z4p"]
Nov 03 00:24:21.266076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.266041    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:21.266339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.266317    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zwnvc\" (UniqueName: \"kubernetes.io/projected/8c9a2bdc-a634-4179-b9ed-3c219dd6b297-kube-api-access-zwnvc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:21.679147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.679105    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"
Nov 03 00:24:21.679147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.679122    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx" event={"ID":"8c9a2bdc-a634-4179-b9ed-3c219dd6b297","Type":"ContainerDied","Data":"45b28bd754c88a6025eafa97581846c3648e74e2b09689071d6cfd7f01b4079d"}
Nov 03 00:24:21.679440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.679194    6000 scope.go:122] "RemoveContainer" containerID="adad5ff269366ac53aef5622573081b67313e481a3fe03352ce90084a079c818"
Nov 03 00:24:21.777349 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.777311    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"]
Nov 03 00:24:21.782344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.781457    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-714512-8555656587-cqcpx"]
Nov 03 00:24:21.869914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.869873    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"]
Nov 03 00:24:21.871541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.871514    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy"
Nov 03 00:24:21.872142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.872120    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy"
Nov 03 00:24:21.872346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.872332    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7b29e82f-0f29-4c92-a07e-c22954e5ce37" containerName="payload-processing"
Nov 03 00:24:21.872462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.872448    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b29e82f-0f29-4c92-a07e-c22954e5ce37" containerName="payload-processing"
Nov 03 00:24:21.872601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.872588    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8c9a2bdc-a634-4179-b9ed-3c219dd6b297" containerName="payload-pre-processing"
Nov 03 00:24:21.872691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.872677    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c9a2bdc-a634-4179-b9ed-3c219dd6b297" containerName="payload-pre-processing"
Nov 03 00:24:21.872782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.872771    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy"
Nov 03 00:24:21.872865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.872854    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy"
Nov 03 00:24:21.873120 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.873102    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c9a2bdc-a634-4179-b9ed-3c219dd6b297" containerName="payload-pre-processing"
Nov 03 00:24:21.873233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.873221    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy"
Nov 03 00:24:21.873387 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.873371    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b29e82f-0f29-4c92-a07e-c22954e5ce37" containerName="payload-processing"
Nov 03 00:24:21.880887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.880860    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:21.935092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.935010    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"]
Nov 03 00:24:21.952613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:21.952582    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-4aa4e9-openshift-default-dockercfg-fkwlw\""
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.101597    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbp4z\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-kube-api-access-mbp4z\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.101747    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.101861    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.101955    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.102039    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.102115    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.102212    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.102241    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.107277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.102292    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205511    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205604    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205637    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205681    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205764    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205877    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205911    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205942    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.206335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.205996    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbp4z\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-kube-api-access-mbp4z\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.259573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.259533    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-kjlpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:24:22.259991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.259955    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:24:22.309283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.260874    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.336371 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.275893    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.336737 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.277897    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.336862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.294729    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.336949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.301544    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.337031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.322650    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.431038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.430917    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.431640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.431616    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.462391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.460964    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbp4z\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-kube-api-access-mbp4z\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.580586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.580547    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:22.649200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.649166    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b29e82f-0f29-4c92-a07e-c22954e5ce37" path="/var/lib/kubelet/pods/7b29e82f-0f29-4c92-a07e-c22954e5ce37/volumes"
Nov 03 00:24:22.650019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:22.649997    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c9a2bdc-a634-4179-b9ed-3c219dd6b297" path="/var/lib/kubelet/pods/8c9a2bdc-a634-4179-b9ed-3c219dd6b297/volumes"
Nov 03 00:24:23.031591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:23.031470    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:24:23.040153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:23.040093    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:24:23.040439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:23.040415    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:24:23.044292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:23.044239    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"]
Nov 03 00:24:23.713782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:23.713747    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" event={"ID":"40eafd36-9dc8-4649-9b0f-b4d13ade4390","Type":"ContainerStarted","Data":"b04201a0975b3b0ed651aa1f0ce72907cbb4e785046ce0fa15656033e1917e80"}
Nov 03 00:24:23.713968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:23.713800    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" event={"ID":"40eafd36-9dc8-4649-9b0f-b4d13ade4390","Type":"ContainerStarted","Data":"248db4723c622cc50f0a76b422add99f5ac9f537f1acfbef2a52feec93df2576"}
Nov 03 00:24:23.753687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:23.753627    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podStartSLOduration=2.753608401 podStartE2EDuration="2.753608401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:23.752188566 +0000 UTC m=+1101.307540590" watchObservedRunningTime="2026-11-03 00:24:23.753608401 +0000 UTC m=+1101.308960437"
Nov 03 00:24:24.588597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:24.588523    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:24.593683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:24.593661    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:24.777291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:24.776926    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:24.778359 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:24.778335    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:24:28.066615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.066584    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"]
Nov 03 00:24:28.067437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.067421    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ffad482-272b-472c-aaa8-0641a2c7d24f" containerName="istio-proxy"
Nov 03 00:24:28.071367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.071352    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.097011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.096980    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-4aa4e9-dockercfg-5k6k9\""
Nov 03 00:24:28.097347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.097300    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-4aa4e9\""
Nov 03 00:24:28.100308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.100236    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"]
Nov 03 00:24:28.133989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.133956    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"]
Nov 03 00:24:28.139488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.139462    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.140961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.140929    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/98eea463-36b2-420d-882d-95e81cc2978e-kube-api-access-vw5q7\") pod \"payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.141134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.141115    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98eea463-36b2-420d-882d-95e81cc2978e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.154546 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.154519    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"]
Nov 03 00:24:28.259284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.252247    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/98eea463-36b2-420d-882d-95e81cc2978e-kube-api-access-vw5q7\") pod \"payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.259284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.252698    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnh76\" (UniqueName: \"kubernetes.io/projected/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-kube-api-access-xnh76\") pod \"payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.259284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.252821    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98eea463-36b2-420d-882d-95e81cc2978e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.259284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.252938    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.259284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.253985    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98eea463-36b2-420d-882d-95e81cc2978e-plugins-config-volume\") pod \"payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.375406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.360229    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnh76\" (UniqueName: \"kubernetes.io/projected/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-kube-api-access-xnh76\") pod \"payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.394801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.394752    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.419381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.407952    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.481406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.480229    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/98eea463-36b2-420d-882d-95e81cc2978e-kube-api-access-vw5q7\") pod \"payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") " pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.481406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.480821    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnh76\" (UniqueName: \"kubernetes.io/projected/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-kube-api-access-xnh76\") pod \"payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.574717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.574681    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"]
Nov 03 00:24:28.575299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.575227    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855" podUID="c88a9e4b-5a0f-4f39-8aea-b65af2b20722" containerName="maas-api" containerID="cri-o://d8e9bf2c59510e975be4695cb85d3124ded2363b13081b0572234c7f23267821" gracePeriod=30
Nov 03 00:24:28.686830 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.686801    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:28.759870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.759830    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:28.836199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.835900    6000 generic.go:356] "Generic (PLEG): container finished" podID="c88a9e4b-5a0f-4f39-8aea-b65af2b20722" containerID="d8e9bf2c59510e975be4695cb85d3124ded2363b13081b0572234c7f23267821" exitCode=0
Nov 03 00:24:28.836199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:28.836105    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855" event={"ID":"c88a9e4b-5a0f-4f39-8aea-b65af2b20722","Type":"ContainerDied","Data":"d8e9bf2c59510e975be4695cb85d3124ded2363b13081b0572234c7f23267821"}
Nov 03 00:24:29.015785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.015752    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"]
Nov 03 00:24:29.021085 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:29.021045    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98eea463_36b2_420d_882d_95e81cc2978e.slice/crio-085c5279a8087640a445a930a8cae6c5f2c55ed2c48f48afa0046376bd431df3 WatchSource:0}: Error finding container 085c5279a8087640a445a930a8cae6c5f2c55ed2c48f48afa0046376bd431df3: Status 404 returned error can't find the container with id 085c5279a8087640a445a930a8cae6c5f2c55ed2c48f48afa0046376bd431df3
Nov 03 00:24:29.343717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.343690    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:24:29.482215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.482176    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-maas-api-tls\") pod \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") "
Nov 03 00:24:29.482399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.482380    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-metrics-certs\") pod \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") "
Nov 03 00:24:29.482470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.482455    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-openshift-service-ca\") pod \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") "
Nov 03 00:24:29.482569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.482548    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-kube-api-access-p9jq4\" (UniqueName: \"kubernetes.io/projected/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-kube-api-access-p9jq4\") pod \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\" (UID: \"c88a9e4b-5a0f-4f39-8aea-b65af2b20722\") "
Nov 03 00:24:29.482831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.482800    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-openshift-service-ca" pod "c88a9e4b-5a0f-4f39-8aea-b65af2b20722" (UID: "c88a9e4b-5a0f-4f39-8aea-b65af2b20722"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:24:29.483365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.483343    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:29.484814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.484775    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-maas-api-tls" pod "c88a9e4b-5a0f-4f39-8aea-b65af2b20722" (UID: "c88a9e4b-5a0f-4f39-8aea-b65af2b20722"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:24:29.484915 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.484813    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-kube-api-access-p9jq4" pod "c88a9e4b-5a0f-4f39-8aea-b65af2b20722" (UID: "c88a9e4b-5a0f-4f39-8aea-b65af2b20722"). InnerVolumeSpecName "kube-api-access-p9jq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:29.505939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.505903    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-metrics-certs" pod "c88a9e4b-5a0f-4f39-8aea-b65af2b20722" (UID: "c88a9e4b-5a0f-4f39-8aea-b65af2b20722"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:24:29.533107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.533079    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"]
Nov 03 00:24:29.534962 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:29.534937    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302a8d35_41ea_4bbf_ba4c_4d0114ff155e.slice/crio-8593a98daac647fc4f34a94dbc1e66b457578662108bc351b2f7bb7c3f4a5f80 WatchSource:0}: Error finding container 8593a98daac647fc4f34a94dbc1e66b457578662108bc351b2f7bb7c3f4a5f80: Status 404 returned error can't find the container with id 8593a98daac647fc4f34a94dbc1e66b457578662108bc351b2f7bb7c3f4a5f80
Nov 03 00:24:29.584814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.584783    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:29.584814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.584812    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:29.584980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.584826    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9jq4\" (UniqueName: \"kubernetes.io/projected/c88a9e4b-5a0f-4f39-8aea-b65af2b20722-kube-api-access-p9jq4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:29.852512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.852472    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855" event={"ID":"c88a9e4b-5a0f-4f39-8aea-b65af2b20722","Type":"ContainerDied","Data":"a334214674e44b2261569310c74f262e1f615fa727c2bfd514f6ed8681ae7176"}
Nov 03 00:24:29.852512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.852515    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"
Nov 03 00:24:29.852844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.852548    6000 scope.go:122] "RemoveContainer" containerID="d8e9bf2c59510e975be4695cb85d3124ded2363b13081b0572234c7f23267821"
Nov 03 00:24:29.855774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.855742    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks" event={"ID":"98eea463-36b2-420d-882d-95e81cc2978e","Type":"ContainerStarted","Data":"14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf"}
Nov 03 00:24:29.855901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.855800    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks" event={"ID":"98eea463-36b2-420d-882d-95e81cc2978e","Type":"ContainerStarted","Data":"085c5279a8087640a445a930a8cae6c5f2c55ed2c48f48afa0046376bd431df3"}
Nov 03 00:24:29.856625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.856602    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:29.858845 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.858818    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf" event={"ID":"302a8d35-41ea-4bbf-ba4c-4d0114ff155e","Type":"ContainerStarted","Data":"a57372172bf30ea363b8332e456de3e99b6cdf694a12f2ba9b44f07d5406f00d"}
Nov 03 00:24:29.858979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.858870    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf" event={"ID":"302a8d35-41ea-4bbf-ba4c-4d0114ff155e","Type":"ContainerStarted","Data":"8593a98daac647fc4f34a94dbc1e66b457578662108bc351b2f7bb7c3f4a5f80"}
Nov 03 00:24:29.858979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.858923    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:29.902178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.902150    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"]
Nov 03 00:24:29.914238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.914209    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-714512-598d787cd8-6m855"]
Nov 03 00:24:29.937742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.937691    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks" podStartSLOduration=1.937674859 podStartE2EDuration="1.937674859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:29.937568469 +0000 UTC m=+1107.492920492" watchObservedRunningTime="2026-11-03 00:24:29.937674859 +0000 UTC m=+1107.493026882"
Nov 03 00:24:29.974269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:29.974213    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf" podStartSLOduration=1.974199794 podStartE2EDuration="1.974199794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:29.972771615 +0000 UTC m=+1107.528123638" watchObservedRunningTime="2026-11-03 00:24:29.974199794 +0000 UTC m=+1107.529551817"
Nov 03 00:24:30.568811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:30.568767    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88a9e4b-5a0f-4f39-8aea-b65af2b20722" path="/var/lib/kubelet/pods/c88a9e4b-5a0f-4f39-8aea-b65af2b20722/volumes"
Nov 03 00:24:37.452652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.452611    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7618-03110e"]
Nov 03 00:24:37.453422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.453405    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c88a9e4b-5a0f-4f39-8aea-b65af2b20722" containerName="maas-api"
Nov 03 00:24:37.453500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.453424    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88a9e4b-5a0f-4f39-8aea-b65af2b20722" containerName="maas-api"
Nov 03 00:24:37.453624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.453611    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88a9e4b-5a0f-4f39-8aea-b65af2b20722" containerName="maas-api"
Nov 03 00:24:37.458512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.458496    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:37.473632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.473608    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7618-03110e"]
Nov 03 00:24:37.481495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.481477    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5fmb\""
Nov 03 00:24:37.552369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.552333    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpr5g\" (UniqueName: \"kubernetes.io/projected/83a06a8f-a816-46b7-a72b-be9c5596ebbd-kube-api-access-zpr5g\") pod \"test-curl-7618-03110e\" (UID: \"83a06a8f-a816-46b7-a72b-be9c5596ebbd\") " pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:37.654217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.654186    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpr5g\" (UniqueName: \"kubernetes.io/projected/83a06a8f-a816-46b7-a72b-be9c5596ebbd-kube-api-access-zpr5g\") pod \"test-curl-7618-03110e\" (UID: \"83a06a8f-a816-46b7-a72b-be9c5596ebbd\") " pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:37.674478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.674452    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpr5g\" (UniqueName: \"kubernetes.io/projected/83a06a8f-a816-46b7-a72b-be9c5596ebbd-kube-api-access-zpr5g\") pod \"test-curl-7618-03110e\" (UID: \"83a06a8f-a816-46b7-a72b-be9c5596ebbd\") " pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:37.781495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:37.770938    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:38.109133 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:38.109102    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod83a06a8f_a816_46b7_a72b_be9c5596ebbd.slice/crio-ebab9f16e7c238df25ff09d596173dfd18cc93c63388752e0042e207d71da85b WatchSource:0}: Error finding container ebab9f16e7c238df25ff09d596173dfd18cc93c63388752e0042e207d71da85b: Status 404 returned error can't find the container with id ebab9f16e7c238df25ff09d596173dfd18cc93c63388752e0042e207d71da85b
Nov 03 00:24:38.109329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:38.109311    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7618-03110e"]
Nov 03 00:24:38.921406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:38.921374    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7618-03110e" event={"ID":"83a06a8f-a816-46b7-a72b-be9c5596ebbd","Type":"ContainerStarted","Data":"ebab9f16e7c238df25ff09d596173dfd18cc93c63388752e0042e207d71da85b"}
Nov 03 00:24:40.877549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:40.873239    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:24:41.875018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:41.874988    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:24:41.949581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:41.949542    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7618-03110e" event={"ID":"83a06a8f-a816-46b7-a72b-be9c5596ebbd","Type":"ContainerStarted","Data":"5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6"}
Nov 03 00:24:41.986810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:41.986763    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7618-03110e" podStartSLOduration=1.942078599 podStartE2EDuration="4.986747781s" totalImagesPullingTime="3.044669182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:41.984912673 +0000 UTC m=+1119.540264697" watchObservedRunningTime="2026-11-03 00:24:41.986747781 +0000 UTC m=+1119.542099804"
Nov 03 00:24:42.221689 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.221608    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"]
Nov 03 00:24:42.226364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.226344    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.241810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.241781    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-4aa4e9-serving-cert\""
Nov 03 00:24:42.245234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.245203    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"]
Nov 03 00:24:42.354923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.354884    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-maas-api-tls\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.355076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.354943    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-metrics-certs\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.355076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.355040    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nv769\" (UniqueName: \"kubernetes.io/projected/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-kube-api-access-nv769\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.355168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.355132    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.462696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.462648    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-maas-api-tls\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.463842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.463785    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-metrics-certs\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.464500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.464479    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nv769\" (UniqueName: \"kubernetes.io/projected/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-kube-api-access-nv769\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.483110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.483032    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.496386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.484033    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-openshift-service-ca\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.506991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.506955    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-maas-api-tls\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.511083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.509032    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nv769\" (UniqueName: \"kubernetes.io/projected/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-kube-api-access-nv769\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.514841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.514807    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-metrics-certs\") pod \"maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.542715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.541910    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:42.875548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.875499    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7618-03110e"]
Nov 03 00:24:42.878717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.878691    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7618-03110e"]
Nov 03 00:24:42.890445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.890407    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"]
Nov 03 00:24:42.890575 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:24:42.890468    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6faf3a50_faf8_4b84_a99f_6f59f8b6140b.slice/crio-9c2413cc2d8c3632f0ff09a8ca8c419c3c3c5119c64fc9d4ff4e6b1d0c57d64c WatchSource:0}: Error finding container 9c2413cc2d8c3632f0ff09a8ca8c419c3c3c5119c64fc9d4ff4e6b1d0c57d64c: Status 404 returned error can't find the container with id 9c2413cc2d8c3632f0ff09a8ca8c419c3c3c5119c64fc9d4ff4e6b1d0c57d64c
Nov 03 00:24:42.957887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:42.957856    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5" event={"ID":"6faf3a50-faf8-4b84-a99f-6f59f8b6140b","Type":"ContainerStarted","Data":"9c2413cc2d8c3632f0ff09a8ca8c419c3c3c5119c64fc9d4ff4e6b1d0c57d64c"}
Nov 03 00:24:43.864194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:43.864162    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:24:44.081285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:44.080740    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7618-03110e" podUID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" containerName="test-curl-7618-03110e" containerID="cri-o://5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6" gracePeriod=2
Nov 03 00:24:44.094316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:44.081780    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5" event={"ID":"6faf3a50-faf8-4b84-a99f-6f59f8b6140b","Type":"ContainerStarted","Data":"109d40da868b8b45c7e2a84693d46bfbb0a2ad2f384303beceefdb0011813633"}
Nov 03 00:24:44.094316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:44.081945    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:44.128619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:44.128512    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5" podStartSLOduration=1.719313632 podStartE2EDuration="2.128496236s" totalImagesPullingTime="409.182604ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:24:44.125475756 +0000 UTC m=+1121.680827782" watchObservedRunningTime="2026-11-03 00:24:44.128496236 +0000 UTC m=+1121.683848260"
Nov 03 00:24:44.137868 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:44.137827    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7618-03110e\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:44.150419 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:44.150381    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7618-03110e\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:46.274547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:46.274526    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:46.338741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:46.338664    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83a06a8f-a816-46b7-a72b-be9c5596ebbd-kube-api-access-zpr5g\" (UniqueName: \"kubernetes.io/projected/83a06a8f-a816-46b7-a72b-be9c5596ebbd-kube-api-access-zpr5g\") pod \"83a06a8f-a816-46b7-a72b-be9c5596ebbd\" (UID: \"83a06a8f-a816-46b7-a72b-be9c5596ebbd\") "
Nov 03 00:24:46.340939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:46.340912    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83a06a8f-a816-46b7-a72b-be9c5596ebbd-kube-api-access-zpr5g" pod "83a06a8f-a816-46b7-a72b-be9c5596ebbd" (UID: "83a06a8f-a816-46b7-a72b-be9c5596ebbd"). InnerVolumeSpecName "kube-api-access-zpr5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:24:46.449373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:46.449337    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpr5g\" (UniqueName: \"kubernetes.io/projected/83a06a8f-a816-46b7-a72b-be9c5596ebbd-kube-api-access-zpr5g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:24:46.567899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:46.567865    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" path="/var/lib/kubelet/pods/83a06a8f-a816-46b7-a72b-be9c5596ebbd/volumes"
Nov 03 00:24:47.105685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:47.105649    6000 generic.go:356] "Generic (PLEG): container finished" podID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" containerID="5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6" exitCode=137
Nov 03 00:24:47.105884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:47.105696    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-03110e"
Nov 03 00:24:47.105884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:47.105729    6000 scope.go:122] "RemoveContainer" containerID="5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6"
Nov 03 00:24:47.115708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:47.115684    6000 scope.go:122] "RemoveContainer" containerID="5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6"
Nov 03 00:24:47.115970 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:47.115949    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6\": container with ID starting with 5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6 not found: ID does not exist" containerID="5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6"
Nov 03 00:24:47.116037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:47.115980    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6"} err="failed to get container status \"5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6\": rpc error: code = NotFound desc = could not find container \"5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6\": container with ID starting with 5efc79ebdf869bf846333a3d04958d80e1454fb2d632027188547dd2cd35f6e6 not found: ID does not exist"
Nov 03 00:24:49.910557 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:24:49.910523    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:24:50.093598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:50.093566    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:24:59.624721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.624680    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"]
Nov 03 00:24:59.625895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.625869    6000 cpu_manager.go:414] "containerMap: removing container" podUID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" containerName="test-curl-7618-03110e"
Nov 03 00:24:59.625997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.625899    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" containerName="test-curl-7618-03110e"
Nov 03 00:24:59.626098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.626084    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="83a06a8f-a816-46b7-a72b-be9c5596ebbd" containerName="test-curl-7618-03110e"
Nov 03 00:24:59.630783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.630763    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.648351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.648322    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"]
Nov 03 00:24:59.657753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.655353    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f3806a-openshift-default-dockercfg-zmnqt\""
Nov 03 00:24:59.685923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.685887    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-data\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.685987    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-credential-socket\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.686064    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-socket\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.686111    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-token\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.686221    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-podinfo\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.686355    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88n2h\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-kube-api-access-88n2h\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686434 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.686403    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-envoy\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.686457    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-certs\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.686551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.686502    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istiod-ca-cert\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.798177    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-podinfo\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821517    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88n2h\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-kube-api-access-88n2h\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821588    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-envoy\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821655    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-certs\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821679    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istiod-ca-cert\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821726    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-data\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821825    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-credential-socket\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821919    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-socket\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.821974    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-token\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.823283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.822367    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-certs\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.825067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.824556    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-data\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.825067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.824786    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-credential-socket\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.825067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.825006    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-socket\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.827579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.827557    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-envoy\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.831360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.831333    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-podinfo\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.831542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.831514    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istiod-ca-cert\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.846134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.846043    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-token\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.858668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.855199    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-88n2h\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-kube-api-access-88n2h\") pod \"e2e-api-a-f3806a-openshift-default-9d7696669-ltd45\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") " pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:24:59.981824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:24:59.981733    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:25:00.623501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:00.623465    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"]
Nov 03 00:25:00.645768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:00.645715    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:00.646236 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:00.645821    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:00.646236 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:00.645869    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:01.317871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:01.317830    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" event={"ID":"b8b6fc11-a279-4997-a8bf-c196b8650d8c","Type":"ContainerStarted","Data":"daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135"}
Nov 03 00:25:01.318209 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:01.317897    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" event={"ID":"b8b6fc11-a279-4997-a8bf-c196b8650d8c","Type":"ContainerStarted","Data":"2493e2cc0b77e66b647b46610ceb755d8b8d94f3b80277bec205b7602a391cb5"}
Nov 03 00:25:01.371117 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:01.371064    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podStartSLOduration=2.371048528 podStartE2EDuration="2.371048528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:01.367808979 +0000 UTC m=+1138.923161013" watchObservedRunningTime="2026-11-03 00:25:01.371048528 +0000 UTC m=+1138.926400553"
Nov 03 00:25:01.982655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:01.982616    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:25:01.998282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:01.990560    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:25:02.326044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:02.325959    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:25:02.326857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:02.326838    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:25:07.039342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.039307    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"]
Nov 03 00:25:07.045929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.045906    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.084169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.084132    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"]
Nov 03 00:25:07.110868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.110834    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjdkr\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-kube-api-access-xjdkr\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.110926    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.111065    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19317f67-59c8-4496-9e87-734482cca794-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.111179    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.111249    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19317f67-59c8-4496-9e87-734482cca794-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.111319    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.111539    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.111583    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.111694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.111675    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.213500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213465    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.213707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213545    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjdkr\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-kube-api-access-xjdkr\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.213707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213613    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.213707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213654    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19317f67-59c8-4496-9e87-734482cca794-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.213875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213710    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.213875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213768    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19317f67-59c8-4496-9e87-734482cca794-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.213875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213799    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.214020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213944    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.214020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.213967    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.214020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.214010    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.214488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.214138    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.214488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.214350    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.214488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.214367    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.214608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.214499    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19317f67-59c8-4496-9e87-734482cca794-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.216551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.216527    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.216705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.216596    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19317f67-59c8-4496-9e87-734482cca794-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.240711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.240684    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.240892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.240875    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjdkr\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-kube-api-access-xjdkr\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.386227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.386176    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:07.713188 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:25:07.713143    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19317f67_59c8_4496_9e87_734482cca794.slice/crio-b277089eaeef0f1da84fb28ebb2635494bb4b2144db749822457f9b373ef6b2b WatchSource:0}: Error finding container b277089eaeef0f1da84fb28ebb2635494bb4b2144db749822457f9b373ef6b2b: Status 404 returned error can't find the container with id b277089eaeef0f1da84fb28ebb2635494bb4b2144db749822457f9b373ef6b2b
Nov 03 00:25:07.714189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.714162    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"]
Nov 03 00:25:07.716004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.715969    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:07.716096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.716034    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:07.716096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:07.716063    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:08.397450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.396153    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" event={"ID":"19317f67-59c8-4496-9e87-734482cca794","Type":"ContainerStarted","Data":"cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b"}
Nov 03 00:25:08.397450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.396225    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" event={"ID":"19317f67-59c8-4496-9e87-734482cca794","Type":"ContainerStarted","Data":"b277089eaeef0f1da84fb28ebb2635494bb4b2144db749822457f9b373ef6b2b"}
Nov 03 00:25:08.400598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.400554    6000 generic.go:356] "Generic (PLEG): container finished" podID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerID="b04201a0975b3b0ed651aa1f0ce72907cbb4e785046ce0fa15656033e1917e80" exitCode=0
Nov 03 00:25:08.400988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.400890    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" event={"ID":"40eafd36-9dc8-4649-9b0f-b4d13ade4390","Type":"ContainerDied","Data":"b04201a0975b3b0ed651aa1f0ce72907cbb4e785046ce0fa15656033e1917e80"}
Nov 03 00:25:08.405953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.405926    6000 scope.go:122] "RemoveContainer" containerID="b04201a0975b3b0ed651aa1f0ce72907cbb4e785046ce0fa15656033e1917e80"
Nov 03 00:25:08.421021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.411495    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:08.421021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.411592    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:08.421021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.411641    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:08.472203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:08.472128    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podStartSLOduration=1.4721106449999999 podStartE2EDuration="1.472110645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:08.460348713 +0000 UTC m=+1146.015700779" watchObservedRunningTime="2026-11-03 00:25:08.472110645 +0000 UTC m=+1146.027462670"
Nov 03 00:25:09.395111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.393582    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:09.397391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.397361    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 00:25:09.397623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.397597    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 00:25:09.441372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.441329    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" event={"ID":"40eafd36-9dc8-4649-9b0f-b4d13ade4390","Type":"ContainerStarted","Data":"6085d7b3bdc8f09b0ba9ba9354e07f0343e48379148a70e6ff0817fd88f08759"}
Nov 03 00:25:09.612312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.610919    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:25:09.612312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.611051    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 00:25:09.612312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.611103    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 00:25:09.777321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:09.777220    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:25:10.409296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.398541    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 00:25:10.409296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.398600    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 00:25:10.600282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.595728    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 00:25:10.600282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.595792    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 00:25:10.954087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.943886    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"]
Nov 03 00:25:10.967958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.967927    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"]
Nov 03 00:25:10.968307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.968285    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:10.983468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:10.983442    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-41e815-openshift-default-dockercfg-sx28t\""
Nov 03 00:25:11.070332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070248    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-token\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070344    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-credential-socket\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070428    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-podinfo\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070533    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq45k\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-kube-api-access-nq45k\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070618    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-data\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070647    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-envoy\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070680    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-certs\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070774    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aecdc9-6974-42f6-b72d-c6da546c6c91-istiod-ca-cert\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.070898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.070801    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-socket\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174220    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-token\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174310    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-credential-socket\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174417    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-podinfo\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174505    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq45k\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-kube-api-access-nq45k\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174606    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-data\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174640    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-envoy\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174703    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-certs\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174840    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aecdc9-6974-42f6-b72d-c6da546c6c91-istiod-ca-cert\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.174865    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-socket\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.175374    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-socket\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.179297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.176339    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-credential-socket\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.233426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.218603    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-certs\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.233426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.219158    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-data\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.233426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.219974    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aecdc9-6974-42f6-b72d-c6da546c6c91-istiod-ca-cert\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.249337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.247173    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq45k\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-kube-api-access-nq45k\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.275083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.275042    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-podinfo\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.280302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.277751    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-token\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.302524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.300912    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-envoy\") pod \"e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") " pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.322602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.322563    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:11.407434 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.404882    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 00:25:11.407434 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.404944    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 00:25:11.601337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.588404    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 00:25:11.601337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.588467    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 00:25:11.627281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:11.621865    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:25:12.040049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.040015    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"]
Nov 03 00:25:12.043684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.043652    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:12.043763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.043717    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:12.043763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.043746    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:12.409714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.409669    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 00:25:12.409928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.409736    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 00:25:12.550647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.550606    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" event={"ID":"93aecdc9-6974-42f6-b72d-c6da546c6c91","Type":"ContainerStarted","Data":"8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b"}
Nov 03 00:25:12.550828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.550695    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" event={"ID":"93aecdc9-6974-42f6-b72d-c6da546c6c91","Type":"ContainerStarted","Data":"8d70e0cdf9096f31d726618a869cc75bf4904c109f81671c2a695dc18b1a7559"}
Nov 03 00:25:12.596279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.591482    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 00:25:12.596279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:12.591543    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 00:25:13.345333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.332750    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:13.362283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.354142    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:13.391279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.390878    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 00:25:13.391279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.390943    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 00:25:13.416282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.408181    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podStartSLOduration=3.408158392 podStartE2EDuration="3.408158392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:12.59457218 +0000 UTC m=+1150.149924205" watchObservedRunningTime="2026-11-03 00:25:13.408158392 +0000 UTC m=+1150.963510418"
Nov 03 00:25:13.636281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.635236    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 00:25:13.636281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.635355    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 00:25:13.640280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.636762    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:13.650280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:13.649945    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:25:14.277566 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:14.277436    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:14.304599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:14.304364    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:14.389280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:14.389093    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 00:25:14.389280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:14.389153    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 00:25:14.604290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:14.601393    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 00:25:14.604290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:14.601470    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 00:25:14.729792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:14.729749    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:15.307282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.306345    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:15.311283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.308087    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:25:15.357618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.357575    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:15.398291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.397418    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 03 00:25:15.398291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.397488    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 03 00:25:15.632305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.618378    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 03 00:25:15.632305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.618443    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 03 00:25:15.763895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:15.763868    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:25:16.403102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:16.403066    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:16.413121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:16.404655    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:16.413121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:16.407039    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:25:16.594171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:16.594142    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:25:16.595539 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:16.595517    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:25:19.702389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.702359    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"]
Nov 03 00:25:19.706707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.706687    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.724946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.724920    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"]
Nov 03 00:25:19.735508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.735467    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-f3806a-serving-cert\""
Nov 03 00:25:19.850673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.850645    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-metrics-certs\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.850872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.850715    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.850959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.850896    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403dedd4-49e9-43de-a391-d0da930a91fe-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.851105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.851084    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnq8\" (UniqueName: \"kubernetes.io/projected/403dedd4-49e9-43de-a391-d0da930a91fe-kube-api-access-lsnq8\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.953635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.953281    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnq8\" (UniqueName: \"kubernetes.io/projected/403dedd4-49e9-43de-a391-d0da930a91fe-kube-api-access-lsnq8\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.953635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.953421    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-metrics-certs\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.953635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.953500    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.953635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.953579    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403dedd4-49e9-43de-a391-d0da930a91fe-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.954380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.954360    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403dedd4-49e9-43de-a391-d0da930a91fe-openshift-service-ca\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.955002 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:19.954876    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-serving-cert: secret "maas-api-e2e-api-a-f3806a-serving-cert" not found
Nov 03 00:25:19.960468 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:19.960449    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls podName:403dedd4-49e9-43de-a391-d0da930a91fe nodeName:}" failed. No retries permitted until 2026-11-03 00:25:20.455226218 +0000 UTC m=+1158.010578221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls") pod "maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6" (UID: "403dedd4-49e9-43de-a391-d0da930a91fe") : secret "maas-api-e2e-api-a-f3806a-serving-cert" not found
Nov 03 00:25:19.997003 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:19.983767    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:25:19.997316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.995132    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnq8\" (UniqueName: \"kubernetes.io/projected/403dedd4-49e9-43de-a391-d0da930a91fe-kube-api-access-lsnq8\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:19.998178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:19.998139    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-metrics-certs\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:20.460988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:20.460956    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:20.463561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:20.463532    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls\") pod \"maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:20.621669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:20.621627    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:21.055446 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:25:21.053662    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403dedd4_49e9_43de_a391_d0da930a91fe.slice/crio-b8c46452c0795e98270ab1ad3f8730239a062a995a84977535b5471f6b1f4fed WatchSource:0}: Error finding container b8c46452c0795e98270ab1ad3f8730239a062a995a84977535b5471f6b1f4fed: Status 404 returned error can't find the container with id b8c46452c0795e98270ab1ad3f8730239a062a995a84977535b5471f6b1f4fed
Nov 03 00:25:21.142029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:21.141991    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"]
Nov 03 00:25:21.900287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:21.899856    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6" event={"ID":"403dedd4-49e9-43de-a391-d0da930a91fe","Type":"ContainerStarted","Data":"fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad"}
Nov 03 00:25:21.900287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:21.899922    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6" event={"ID":"403dedd4-49e9-43de-a391-d0da930a91fe","Type":"ContainerStarted","Data":"b8c46452c0795e98270ab1ad3f8730239a062a995a84977535b5471f6b1f4fed"}
Nov 03 00:25:21.902359 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:21.901570    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:21.943081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:21.943028    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6" podStartSLOduration=2.590305924 podStartE2EDuration="2.943013476s" totalImagesPullingTime="352.707552ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:21.93887771 +0000 UTC m=+1159.494229734" watchObservedRunningTime="2026-11-03 00:25:21.943013476 +0000 UTC m=+1159.498365501"
Nov 03 00:25:25.975949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:25.975911    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"]
Nov 03 00:25:26.007373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:26.007331    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"]
Nov 03 00:25:27.083970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.083933    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"]
Nov 03 00:25:27.084465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.084133    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2" podUID="0fec23fd-b176-4386-a861-277c2dde2591" containerName="maas-api" containerID="cri-o://dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5" gracePeriod=30
Nov 03 00:25:27.372667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.372641    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:25:27.427396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.427366    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-metrics-certs\") pod \"0fec23fd-b176-4386-a861-277c2dde2591\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") "
Nov 03 00:25:27.427587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.427473    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fec23fd-b176-4386-a861-277c2dde2591-kube-api-access-6xdzv\" (UniqueName: \"kubernetes.io/projected/0fec23fd-b176-4386-a861-277c2dde2591-kube-api-access-6xdzv\") pod \"0fec23fd-b176-4386-a861-277c2dde2591\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") "
Nov 03 00:25:27.427587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.427533    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-maas-api-tls\") pod \"0fec23fd-b176-4386-a861-277c2dde2591\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") "
Nov 03 00:25:27.427587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.427575    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fec23fd-b176-4386-a861-277c2dde2591-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec23fd-b176-4386-a861-277c2dde2591-openshift-service-ca\") pod \"0fec23fd-b176-4386-a861-277c2dde2591\" (UID: \"0fec23fd-b176-4386-a861-277c2dde2591\") "
Nov 03 00:25:27.428227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.428194    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fec23fd-b176-4386-a861-277c2dde2591-openshift-service-ca" pod "0fec23fd-b176-4386-a861-277c2dde2591" (UID: "0fec23fd-b176-4386-a861-277c2dde2591"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:27.428456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.428433    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fec23fd-b176-4386-a861-277c2dde2591-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:27.430521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.430481    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-maas-api-tls" pod "0fec23fd-b176-4386-a861-277c2dde2591" (UID: "0fec23fd-b176-4386-a861-277c2dde2591"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:27.430661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.430639    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-metrics-certs" pod "0fec23fd-b176-4386-a861-277c2dde2591" (UID: "0fec23fd-b176-4386-a861-277c2dde2591"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:27.430725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.430657    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fec23fd-b176-4386-a861-277c2dde2591-kube-api-access-6xdzv" pod "0fec23fd-b176-4386-a861-277c2dde2591" (UID: "0fec23fd-b176-4386-a861-277c2dde2591"). InnerVolumeSpecName "kube-api-access-6xdzv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:27.587285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.547805    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:27.587285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.547844    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xdzv\" (UniqueName: \"kubernetes.io/projected/0fec23fd-b176-4386-a861-277c2dde2591-kube-api-access-6xdzv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:27.587285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.547866    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fec23fd-b176-4386-a861-277c2dde2591-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:27.952494 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.952463    6000 generic.go:356] "Generic (PLEG): container finished" podID="0fec23fd-b176-4386-a861-277c2dde2591" containerID="dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5" exitCode=0
Nov 03 00:25:27.952673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.952510    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"
Nov 03 00:25:27.952673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.952546    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2" event={"ID":"0fec23fd-b176-4386-a861-277c2dde2591","Type":"ContainerDied","Data":"dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5"}
Nov 03 00:25:27.952673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.952599    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2" event={"ID":"0fec23fd-b176-4386-a861-277c2dde2591","Type":"ContainerDied","Data":"88e1fb95ea71477ef9a480b714390bc846e9a3892eb6944fc2ca9c98313510a7"}
Nov 03 00:25:27.952673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.952623    6000 scope.go:122] "RemoveContainer" containerID="dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5"
Nov 03 00:25:27.968138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.968114    6000 scope.go:122] "RemoveContainer" containerID="dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5"
Nov 03 00:25:27.968485 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:27.968464    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5\": container with ID starting with dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5 not found: ID does not exist" containerID="dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5"
Nov 03 00:25:27.968580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:27.968497    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5"} err="failed to get container status \"dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5\": rpc error: code = NotFound desc = could not find container \"dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5\": container with ID starting with dd4463bb7c69abb932d76731f2ab90cfaa84d377a2799726450cad7b4f988bb5 not found: ID does not exist"
Nov 03 00:25:28.007585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:28.007554    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"]
Nov 03 00:25:28.049290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:28.034946    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-5c5686-74d6fcfb46-5kkc2"]
Nov 03 00:25:28.568905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:28.568862    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fec23fd-b176-4386-a861-277c2dde2591" path="/var/lib/kubelet/pods/0fec23fd-b176-4386-a861-277c2dde2591/volumes"
Nov 03 00:25:28.918587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:28.918561    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:25:30.977990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:30.977936    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7" podUID="1abd54d8-38bb-405a-8ed8-31b556eb33ae" containerName="payload-processing" containerID="cri-o://d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a" gracePeriod=25
Nov 03 00:25:31.028004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.027966    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj" podUID="3d1a58a2-1ce7-4c6f-930f-b084ef702624" containerName="payload-pre-processing" containerID="cri-o://2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9" gracePeriod=25
Nov 03 00:25:31.386979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.386946    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:25:31.440526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.438309    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1abd54d8-38bb-405a-8ed8-31b556eb33ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1abd54d8-38bb-405a-8ed8-31b556eb33ae-plugins-config-volume\") pod \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") "
Nov 03 00:25:31.440526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.438439    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1abd54d8-38bb-405a-8ed8-31b556eb33ae-kube-api-access-fxxlk\" (UniqueName: \"kubernetes.io/projected/1abd54d8-38bb-405a-8ed8-31b556eb33ae-kube-api-access-fxxlk\") pod \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\" (UID: \"1abd54d8-38bb-405a-8ed8-31b556eb33ae\") "
Nov 03 00:25:31.440526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.438872    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1abd54d8-38bb-405a-8ed8-31b556eb33ae-plugins-config-volume" pod "1abd54d8-38bb-405a-8ed8-31b556eb33ae" (UID: "1abd54d8-38bb-405a-8ed8-31b556eb33ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:31.440526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.439747    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1abd54d8-38bb-405a-8ed8-31b556eb33ae-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:31.441733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.441708    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1abd54d8-38bb-405a-8ed8-31b556eb33ae-kube-api-access-fxxlk" pod "1abd54d8-38bb-405a-8ed8-31b556eb33ae" (UID: "1abd54d8-38bb-405a-8ed8-31b556eb33ae"). InnerVolumeSpecName "kube-api-access-fxxlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:31.452456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.452435    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:25:31.541151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.541057    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d1a58a2-1ce7-4c6f-930f-b084ef702624-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1a58a2-1ce7-4c6f-930f-b084ef702624-plugins-config-volume\") pod \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") "
Nov 03 00:25:31.541335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.541314    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d1a58a2-1ce7-4c6f-930f-b084ef702624-kube-api-access-4zjw2\" (UniqueName: \"kubernetes.io/projected/3d1a58a2-1ce7-4c6f-930f-b084ef702624-kube-api-access-4zjw2\") pod \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\" (UID: \"3d1a58a2-1ce7-4c6f-930f-b084ef702624\") "
Nov 03 00:25:31.541520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.541483    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d1a58a2-1ce7-4c6f-930f-b084ef702624-plugins-config-volume" pod "3d1a58a2-1ce7-4c6f-930f-b084ef702624" (UID: "3d1a58a2-1ce7-4c6f-930f-b084ef702624"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:31.542011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.541991    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d1a58a2-1ce7-4c6f-930f-b084ef702624-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:31.542115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.542018    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxxlk\" (UniqueName: \"kubernetes.io/projected/1abd54d8-38bb-405a-8ed8-31b556eb33ae-kube-api-access-fxxlk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:31.543794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.543768    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d1a58a2-1ce7-4c6f-930f-b084ef702624-kube-api-access-4zjw2" pod "3d1a58a2-1ce7-4c6f-930f-b084ef702624" (UID: "3d1a58a2-1ce7-4c6f-930f-b084ef702624"). InnerVolumeSpecName "kube-api-access-4zjw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:31.643933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:31.643899    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zjw2\" (UniqueName: \"kubernetes.io/projected/3d1a58a2-1ce7-4c6f-930f-b084ef702624-kube-api-access-4zjw2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:32.026279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.025647    6000 generic.go:356] "Generic (PLEG): container finished" podID="1abd54d8-38bb-405a-8ed8-31b556eb33ae" containerID="d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a" exitCode=0
Nov 03 00:25:32.026279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.025766    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7" event={"ID":"1abd54d8-38bb-405a-8ed8-31b556eb33ae","Type":"ContainerDied","Data":"d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a"}
Nov 03 00:25:32.026279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.025826    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7" event={"ID":"1abd54d8-38bb-405a-8ed8-31b556eb33ae","Type":"ContainerDied","Data":"174b1f55057bdd22b6169b9115d445d8f3ddc2cf1252dc8b04716fdae7dbb7f5"}
Nov 03 00:25:32.026279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.025861    6000 scope.go:122] "RemoveContainer" containerID="d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a"
Nov 03 00:25:32.026279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.026013    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"
Nov 03 00:25:32.033763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.030728    6000 generic.go:356] "Generic (PLEG): container finished" podID="3d1a58a2-1ce7-4c6f-930f-b084ef702624" containerID="2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9" exitCode=0
Nov 03 00:25:32.033763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.030819    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj" event={"ID":"3d1a58a2-1ce7-4c6f-930f-b084ef702624","Type":"ContainerDied","Data":"2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9"}
Nov 03 00:25:32.033763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.030899    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj" event={"ID":"3d1a58a2-1ce7-4c6f-930f-b084ef702624","Type":"ContainerDied","Data":"6823f48dd8a6f83839443be5f946915d7c04978ffea68ac20df2e24095ed562a"}
Nov 03 00:25:32.033763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.030974    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"
Nov 03 00:25:32.048049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.048028    6000 scope.go:122] "RemoveContainer" containerID="d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a"
Nov 03 00:25:32.048394 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:32.048372    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a\": container with ID starting with d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a not found: ID does not exist" containerID="d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a"
Nov 03 00:25:32.048471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.048403    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a"} err="failed to get container status \"d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a\": rpc error: code = NotFound desc = could not find container \"d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a\": container with ID starting with d3029933fbf48669c54d63743332151a06e5fcb78acdb327017f427c7c15af7a not found: ID does not exist"
Nov 03 00:25:32.048471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.048422    6000 scope.go:122] "RemoveContainer" containerID="2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9"
Nov 03 00:25:32.057708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.057689    6000 scope.go:122] "RemoveContainer" containerID="2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9"
Nov 03 00:25:32.058010 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:32.057992    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9\": container with ID starting with 2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9 not found: ID does not exist" containerID="2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9"
Nov 03 00:25:32.058054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.058023    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9"} err="failed to get container status \"2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9\": rpc error: code = NotFound desc = could not find container \"2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9\": container with ID starting with 2ddb7afdfe9cf2286dfd96583e3c3c53edf633157c6b422d1d643fd69c3c46e9 not found: ID does not exist"
Nov 03 00:25:32.102101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.102074    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"]
Nov 03 00:25:32.113202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.113175    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9f76b708-6868dbb7f7-dvns7"]
Nov 03 00:25:32.145156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.145124    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"]
Nov 03 00:25:32.153220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.153192    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9f76b708-867f4ccfb7-sr2gj"]
Nov 03 00:25:32.571067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.571034    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1abd54d8-38bb-405a-8ed8-31b556eb33ae" path="/var/lib/kubelet/pods/1abd54d8-38bb-405a-8ed8-31b556eb33ae/volumes"
Nov 03 00:25:32.571600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.571583    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d1a58a2-1ce7-4c6f-930f-b084ef702624" path="/var/lib/kubelet/pods/3d1a58a2-1ce7-4c6f-930f-b084ef702624/volumes"
Nov 03 00:25:32.733784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.733751    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"]
Nov 03 00:25:32.734655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734638    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0fec23fd-b176-4386-a861-277c2dde2591" containerName="maas-api"
Nov 03 00:25:32.734727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734658    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fec23fd-b176-4386-a861-277c2dde2591" containerName="maas-api"
Nov 03 00:25:32.734727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734671    6000 cpu_manager.go:414] "containerMap: removing container" podUID="3d1a58a2-1ce7-4c6f-930f-b084ef702624" containerName="payload-pre-processing"
Nov 03 00:25:32.734727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734676    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d1a58a2-1ce7-4c6f-930f-b084ef702624" containerName="payload-pre-processing"
Nov 03 00:25:32.734727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734694    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1abd54d8-38bb-405a-8ed8-31b556eb33ae" containerName="payload-processing"
Nov 03 00:25:32.734727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734699    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1abd54d8-38bb-405a-8ed8-31b556eb33ae" containerName="payload-processing"
Nov 03 00:25:32.734880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734812    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fec23fd-b176-4386-a861-277c2dde2591" containerName="maas-api"
Nov 03 00:25:32.734880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734822    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d1a58a2-1ce7-4c6f-930f-b084ef702624" containerName="payload-pre-processing"
Nov 03 00:25:32.734880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.734830    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1abd54d8-38bb-405a-8ed8-31b556eb33ae" containerName="payload-processing"
Nov 03 00:25:32.740293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.740239    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:32.757567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.757541    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-493ec8\""
Nov 03 00:25:32.761985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.761954    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"]
Nov 03 00:25:32.766755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.766731    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"]
Nov 03 00:25:32.771599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.771579    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:32.788250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.788226    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-493ec8-dockercfg-vfl65\""
Nov 03 00:25:32.788568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.788539    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"]
Nov 03 00:25:32.947280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.946134    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6765be6-efe8-457f-be3d-3e2fd547684d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") " pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:32.947280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.946298    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:32.947280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.946353    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6chh\" (UniqueName: \"kubernetes.io/projected/a6765be6-efe8-457f-be3d-3e2fd547684d-kube-api-access-d6chh\") pod \"payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") " pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:32.947280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:32.946382    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5ks8\" (UniqueName: \"kubernetes.io/projected/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-kube-api-access-l5ks8\") pod \"payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:33.068289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.064224    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6765be6-efe8-457f-be3d-3e2fd547684d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") " pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:33.068289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.064388    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:33.068289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.064442    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6chh\" (UniqueName: \"kubernetes.io/projected/a6765be6-efe8-457f-be3d-3e2fd547684d-kube-api-access-d6chh\") pod \"payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") " pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:33.068289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.064474    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5ks8\" (UniqueName: \"kubernetes.io/projected/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-kube-api-access-l5ks8\") pod \"payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:33.068289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.066082    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6765be6-efe8-457f-be3d-3e2fd547684d-plugins-config-volume\") pod \"payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") " pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:33.068289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.066831    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:33.105280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.102071    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6chh\" (UniqueName: \"kubernetes.io/projected/a6765be6-efe8-457f-be3d-3e2fd547684d-kube-api-access-d6chh\") pod \"payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") " pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:33.130284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.129470    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5ks8\" (UniqueName: \"kubernetes.io/projected/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-kube-api-access-l5ks8\") pod \"payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:33.358381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.358349    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:33.392627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.392594    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:33.663565 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:25:33.663533    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6765be6_efe8_457f_be3d_3e2fd547684d.slice/crio-168e4b9a3526c034ce0d0348afa2887ff20676242b15f6c28b10af8646f4d544 WatchSource:0}: Error finding container 168e4b9a3526c034ce0d0348afa2887ff20676242b15f6c28b10af8646f4d544: Status 404 returned error can't find the container with id 168e4b9a3526c034ce0d0348afa2887ff20676242b15f6c28b10af8646f4d544
Nov 03 00:25:33.664040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.664016    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"]
Nov 03 00:25:33.718552 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:25:33.718522    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aba12fe_31b5_4eba_b3be_fdccd6b3cedf.slice/crio-c6e2d8df581c381f44eea424008176c9437147542082e00cf8ce6a2f34c625b2 WatchSource:0}: Error finding container c6e2d8df581c381f44eea424008176c9437147542082e00cf8ce6a2f34c625b2: Status 404 returned error can't find the container with id c6e2d8df581c381f44eea424008176c9437147542082e00cf8ce6a2f34c625b2
Nov 03 00:25:33.718939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.718918    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"]
Nov 03 00:25:33.877345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.877312    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"]
Nov 03 00:25:33.882003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.881981    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:33.909701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.909624    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-41e815-serving-cert\""
Nov 03 00:25:33.913365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.913314    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"]
Nov 03 00:25:33.980225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.980184    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:33.980442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.980359    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-metrics-certs\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:33.980517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.980491    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad28b7c-811a-4225-a1b5-de0a3138978e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:33.980630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:33.980616    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrpdx\" (UniqueName: \"kubernetes.io/projected/fad28b7c-811a-4225-a1b5-de0a3138978e-kube-api-access-nrpdx\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.081697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.081665    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrpdx\" (UniqueName: \"kubernetes.io/projected/fad28b7c-811a-4225-a1b5-de0a3138978e-kube-api-access-nrpdx\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.082144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.081822    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.082144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.081863    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-metrics-certs\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.082144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.081914    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad28b7c-811a-4225-a1b5-de0a3138978e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.082588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.082564    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad28b7c-811a-4225-a1b5-de0a3138978e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.084668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.084640    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.084792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.084771    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-metrics-certs\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.103399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.103356    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrpdx\" (UniqueName: \"kubernetes.io/projected/fad28b7c-811a-4225-a1b5-de0a3138978e-kube-api-access-nrpdx\") pod \"maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.113014 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.112981    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r" event={"ID":"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf","Type":"ContainerStarted","Data":"71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d"}
Nov 03 00:25:34.113149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.113046    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:34.113149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.113067    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r" event={"ID":"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf","Type":"ContainerStarted","Data":"c6e2d8df581c381f44eea424008176c9437147542082e00cf8ce6a2f34c625b2"}
Nov 03 00:25:34.114694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.114672    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z" event={"ID":"a6765be6-efe8-457f-be3d-3e2fd547684d","Type":"ContainerStarted","Data":"ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59"}
Nov 03 00:25:34.114822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.114709    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z" event={"ID":"a6765be6-efe8-457f-be3d-3e2fd547684d","Type":"ContainerStarted","Data":"168e4b9a3526c034ce0d0348afa2887ff20676242b15f6c28b10af8646f4d544"}
Nov 03 00:25:34.114822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.114780    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:34.151130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.151068    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r" podStartSLOduration=2.151054744 podStartE2EDuration="2.151054744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:34.148508283 +0000 UTC m=+1171.703860309" watchObservedRunningTime="2026-11-03 00:25:34.151054744 +0000 UTC m=+1171.706406768"
Nov 03 00:25:34.185903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.185806    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z" podStartSLOduration=2.185793088 podStartE2EDuration="2.185793088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:34.185456588 +0000 UTC m=+1171.740808614" watchObservedRunningTime="2026-11-03 00:25:34.185793088 +0000 UTC m=+1171.741145111"
Nov 03 00:25:34.195937 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.195911    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:34.521502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:34.521469    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"]
Nov 03 00:25:34.523302 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:25:34.523270    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad28b7c_811a_4225_a1b5_de0a3138978e.slice/crio-a7fa93cc40639fe7eda167be898900333688659f6d0e86bb387490a88e814f8a WatchSource:0}: Error finding container a7fa93cc40639fe7eda167be898900333688659f6d0e86bb387490a88e814f8a: Status 404 returned error can't find the container with id a7fa93cc40639fe7eda167be898900333688659f6d0e86bb387490a88e814f8a
Nov 03 00:25:35.122779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:35.122744    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn" event={"ID":"fad28b7c-811a-4225-a1b5-de0a3138978e","Type":"ContainerStarted","Data":"2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2"}
Nov 03 00:25:35.123278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:35.122801    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn" event={"ID":"fad28b7c-811a-4225-a1b5-de0a3138978e","Type":"ContainerStarted","Data":"a7fa93cc40639fe7eda167be898900333688659f6d0e86bb387490a88e814f8a"}
Nov 03 00:25:35.123278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:35.123072    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:35.277860 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:35.199783    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn" podStartSLOduration=1.837225603 podStartE2EDuration="2.199762457s" totalImagesPullingTime="362.536854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:35.198627456 +0000 UTC m=+1172.753979483" watchObservedRunningTime="2026-11-03 00:25:35.199762457 +0000 UTC m=+1172.755114482"
Nov 03 00:25:37.786967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:37.786927    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"]
Nov 03 00:25:37.866436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:37.866402    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"]
Nov 03 00:25:41.177756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:41.177729    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:25:41.666926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:41.666894    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"]
Nov 03 00:25:41.667166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:41.667140    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6" podUID="6ac20e9e-db09-46ae-9721-077f93df6b9c" containerName="maas-api" containerID="cri-o://e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22" gracePeriod=30
Nov 03 00:25:42.039332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.030788    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:25:42.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.116800    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-maas-api-tls\") pod \"6ac20e9e-db09-46ae-9721-077f93df6b9c\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") "
Nov 03 00:25:42.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.116886    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ac20e9e-db09-46ae-9721-077f93df6b9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac20e9e-db09-46ae-9721-077f93df6b9c-openshift-service-ca\") pod \"6ac20e9e-db09-46ae-9721-077f93df6b9c\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") "
Nov 03 00:25:42.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.117099    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ac20e9e-db09-46ae-9721-077f93df6b9c-kube-api-access-7cs6j\" (UniqueName: \"kubernetes.io/projected/6ac20e9e-db09-46ae-9721-077f93df6b9c-kube-api-access-7cs6j\") pod \"6ac20e9e-db09-46ae-9721-077f93df6b9c\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") "
Nov 03 00:25:42.117629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.117323    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-metrics-certs\") pod \"6ac20e9e-db09-46ae-9721-077f93df6b9c\" (UID: \"6ac20e9e-db09-46ae-9721-077f93df6b9c\") "
Nov 03 00:25:42.121279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.118953    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ac20e9e-db09-46ae-9721-077f93df6b9c-openshift-service-ca" pod "6ac20e9e-db09-46ae-9721-077f93df6b9c" (UID: "6ac20e9e-db09-46ae-9721-077f93df6b9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:42.155470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.155425    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ac20e9e-db09-46ae-9721-077f93df6b9c-kube-api-access-7cs6j" pod "6ac20e9e-db09-46ae-9721-077f93df6b9c" (UID: "6ac20e9e-db09-46ae-9721-077f93df6b9c"). InnerVolumeSpecName "kube-api-access-7cs6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:42.228658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.225764    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-maas-api-tls" pod "6ac20e9e-db09-46ae-9721-077f93df6b9c" (UID: "6ac20e9e-db09-46ae-9721-077f93df6b9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:42.228658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.225865    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-metrics-certs" pod "6ac20e9e-db09-46ae-9721-077f93df6b9c" (UID: "6ac20e9e-db09-46ae-9721-077f93df6b9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:42.229202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.229104    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cs6j\" (UniqueName: \"kubernetes.io/projected/6ac20e9e-db09-46ae-9721-077f93df6b9c-kube-api-access-7cs6j\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:42.229202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.229132    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:42.229202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.229145    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ac20e9e-db09-46ae-9721-077f93df6b9c-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:42.229202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.229157    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ac20e9e-db09-46ae-9721-077f93df6b9c-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:42.281643 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.281600    6000 generic.go:356] "Generic (PLEG): container finished" podID="6ac20e9e-db09-46ae-9721-077f93df6b9c" containerID="e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22" exitCode=0
Nov 03 00:25:42.281844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.281820    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6" event={"ID":"6ac20e9e-db09-46ae-9721-077f93df6b9c","Type":"ContainerDied","Data":"e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22"}
Nov 03 00:25:42.281913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.281884    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6" event={"ID":"6ac20e9e-db09-46ae-9721-077f93df6b9c","Type":"ContainerDied","Data":"7cb5153194545a43319f59a7695aef71092b05ad9b0869985481069f68369a37"}
Nov 03 00:25:42.281971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.281916    6000 scope.go:122] "RemoveContainer" containerID="e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22"
Nov 03 00:25:42.282101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.282082    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"
Nov 03 00:25:42.395959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.395919    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"]
Nov 03 00:25:42.397086 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.397042    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-9f76b708-556c4c7c7-xmlh6"]
Nov 03 00:25:42.427354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.426943    6000 scope.go:122] "RemoveContainer" containerID="e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22"
Nov 03 00:25:42.464332 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:42.457761    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22\": container with ID starting with e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22 not found: ID does not exist" containerID="e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22"
Nov 03 00:25:42.464332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.457813    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22"} err="failed to get container status \"e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22\": rpc error: code = NotFound desc = could not find container \"e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22\": container with ID starting with e67e135693bf333d07c5835a561257df6564b93214ead903a70c9dd9a53a6f22 not found: ID does not exist"
Nov 03 00:25:42.631279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.629980    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ac20e9e-db09-46ae-9721-077f93df6b9c" path="/var/lib/kubelet/pods/6ac20e9e-db09-46ae-9721-077f93df6b9c/volumes"
Nov 03 00:25:42.787485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.787343    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks" podUID="98eea463-36b2-420d-882d-95e81cc2978e" containerName="payload-processing" containerID="cri-o://14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf" gracePeriod=25
Nov 03 00:25:42.867202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:42.867145    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf" podUID="302a8d35-41ea-4bbf-ba4c-4d0114ff155e" containerName="payload-pre-processing" containerID="cri-o://a57372172bf30ea363b8332e456de3e99b6cdf694a12f2ba9b44f07d5406f00d" gracePeriod=25
Nov 03 00:25:43.273295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.270912    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:25:43.322355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.322050    6000 generic.go:356] "Generic (PLEG): container finished" podID="302a8d35-41ea-4bbf-ba4c-4d0114ff155e" containerID="a57372172bf30ea363b8332e456de3e99b6cdf694a12f2ba9b44f07d5406f00d" exitCode=0
Nov 03 00:25:43.322355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.322120    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf" event={"ID":"302a8d35-41ea-4bbf-ba4c-4d0114ff155e","Type":"ContainerDied","Data":"a57372172bf30ea363b8332e456de3e99b6cdf694a12f2ba9b44f07d5406f00d"}
Nov 03 00:25:43.324445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.324418    6000 generic.go:356] "Generic (PLEG): container finished" podID="98eea463-36b2-420d-882d-95e81cc2978e" containerID="14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf" exitCode=0
Nov 03 00:25:43.324655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.324499    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"
Nov 03 00:25:43.325188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.324522    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks" event={"ID":"98eea463-36b2-420d-882d-95e81cc2978e","Type":"ContainerDied","Data":"14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf"}
Nov 03 00:25:43.325188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.324840    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks" event={"ID":"98eea463-36b2-420d-882d-95e81cc2978e","Type":"ContainerDied","Data":"085c5279a8087640a445a930a8cae6c5f2c55ed2c48f48afa0046376bd431df3"}
Nov 03 00:25:43.325188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.324876    6000 scope.go:122] "RemoveContainer" containerID="14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf"
Nov 03 00:25:43.337805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.337782    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:25:43.338535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.338517    6000 scope.go:122] "RemoveContainer" containerID="14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf"
Nov 03 00:25:43.338873 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:43.338847    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf\": container with ID starting with 14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf not found: ID does not exist" containerID="14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf"
Nov 03 00:25:43.339028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.339002    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf"} err="failed to get container status \"14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf\": rpc error: code = NotFound desc = could not find container \"14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf\": container with ID starting with 14b7fc84b341871b997098e9db354642a537a63c10b7f4f88de1bcd1a8e6eccf not found: ID does not exist"
Nov 03 00:25:43.419377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.418555    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98eea463-36b2-420d-882d-95e81cc2978e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98eea463-36b2-420d-882d-95e81cc2978e-plugins-config-volume\") pod \"98eea463-36b2-420d-882d-95e81cc2978e\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") "
Nov 03 00:25:43.419377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.418639    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98eea463-36b2-420d-882d-95e81cc2978e-kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/98eea463-36b2-420d-882d-95e81cc2978e-kube-api-access-vw5q7\") pod \"98eea463-36b2-420d-882d-95e81cc2978e\" (UID: \"98eea463-36b2-420d-882d-95e81cc2978e\") "
Nov 03 00:25:43.420195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.420161    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98eea463-36b2-420d-882d-95e81cc2978e-plugins-config-volume" pod "98eea463-36b2-420d-882d-95e81cc2978e" (UID: "98eea463-36b2-420d-882d-95e81cc2978e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:43.422893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.422862    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98eea463-36b2-420d-882d-95e81cc2978e-kube-api-access-vw5q7" pod "98eea463-36b2-420d-882d-95e81cc2978e" (UID: "98eea463-36b2-420d-882d-95e81cc2978e"). InnerVolumeSpecName "kube-api-access-vw5q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:43.520573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.520535    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-plugins-config-volume\") pod \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") "
Nov 03 00:25:43.520773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.520668    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-kube-api-access-xnh76\" (UniqueName: \"kubernetes.io/projected/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-kube-api-access-xnh76\") pod \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\" (UID: \"302a8d35-41ea-4bbf-ba4c-4d0114ff155e\") "
Nov 03 00:25:43.520975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.520940    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-plugins-config-volume" pod "302a8d35-41ea-4bbf-ba4c-4d0114ff155e" (UID: "302a8d35-41ea-4bbf-ba4c-4d0114ff155e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:43.521457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.521437    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:43.521547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.521460    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/98eea463-36b2-420d-882d-95e81cc2978e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:43.521547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.521471    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5q7\" (UniqueName: \"kubernetes.io/projected/98eea463-36b2-420d-882d-95e81cc2978e-kube-api-access-vw5q7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:43.523062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.523040    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-kube-api-access-xnh76" pod "302a8d35-41ea-4bbf-ba4c-4d0114ff155e" (UID: "302a8d35-41ea-4bbf-ba4c-4d0114ff155e"). InnerVolumeSpecName "kube-api-access-xnh76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:43.623132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.623095    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnh76\" (UniqueName: \"kubernetes.io/projected/302a8d35-41ea-4bbf-ba4c-4d0114ff155e-kube-api-access-xnh76\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:43.671867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.671834    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"]
Nov 03 00:25:43.675735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:43.675702    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-4aa4e9-646cc5868f-rdgks"]
Nov 03 00:25:44.336316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:44.336288    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"
Nov 03 00:25:44.336838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:44.336299    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf" event={"ID":"302a8d35-41ea-4bbf-ba4c-4d0114ff155e","Type":"ContainerDied","Data":"8593a98daac647fc4f34a94dbc1e66b457578662108bc351b2f7bb7c3f4a5f80"}
Nov 03 00:25:44.336959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:44.336899    6000 scope.go:122] "RemoveContainer" containerID="a57372172bf30ea363b8332e456de3e99b6cdf694a12f2ba9b44f07d5406f00d"
Nov 03 00:25:44.409277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:44.395601    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"]
Nov 03 00:25:44.430282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:44.424188    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-4aa4e9-6c74487659-vrcnf"]
Nov 03 00:25:44.568940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:44.568904    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302a8d35-41ea-4bbf-ba4c-4d0114ff155e" path="/var/lib/kubelet/pods/302a8d35-41ea-4bbf-ba4c-4d0114ff155e/volumes"
Nov 03 00:25:44.569420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:44.569404    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98eea463-36b2-420d-882d-95e81cc2978e" path="/var/lib/kubelet/pods/98eea463-36b2-420d-882d-95e81cc2978e/volumes"
Nov 03 00:25:45.126617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:45.126588    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:25:45.126796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:45.126675    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:25:48.913882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.913847    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"]
Nov 03 00:25:48.914738 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.914717    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6ac20e9e-db09-46ae-9721-077f93df6b9c" containerName="maas-api"
Nov 03 00:25:48.914839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.914740    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ac20e9e-db09-46ae-9721-077f93df6b9c" containerName="maas-api"
Nov 03 00:25:48.914839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.914775    6000 cpu_manager.go:414] "containerMap: removing container" podUID="98eea463-36b2-420d-882d-95e81cc2978e" containerName="payload-processing"
Nov 03 00:25:48.914839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.914784    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98eea463-36b2-420d-882d-95e81cc2978e" containerName="payload-processing"
Nov 03 00:25:48.915004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.914843    6000 cpu_manager.go:414] "containerMap: removing container" podUID="302a8d35-41ea-4bbf-ba4c-4d0114ff155e" containerName="payload-pre-processing"
Nov 03 00:25:48.915004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.914851    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302a8d35-41ea-4bbf-ba4c-4d0114ff155e" containerName="payload-pre-processing"
Nov 03 00:25:48.915100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.915021    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="302a8d35-41ea-4bbf-ba4c-4d0114ff155e" containerName="payload-pre-processing"
Nov 03 00:25:48.915100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.915039    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ac20e9e-db09-46ae-9721-077f93df6b9c" containerName="maas-api"
Nov 03 00:25:48.915100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.915051    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="98eea463-36b2-420d-882d-95e81cc2978e" containerName="payload-processing"
Nov 03 00:25:48.918531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.918513    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:48.935326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.935304    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-493ec8-serving-cert\""
Nov 03 00:25:48.950882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.950841    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7014038a-0215-4164-b032-feff0b68056f-openshift-service-ca\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:48.951302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.951248    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"]
Nov 03 00:25:48.951537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.951177    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gbnq\" (UniqueName: \"kubernetes.io/projected/7014038a-0215-4164-b032-feff0b68056f-kube-api-access-6gbnq\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:48.951693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.951672    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-metrics-certs\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:48.951820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:48.951726    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.057708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.057677    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7014038a-0215-4164-b032-feff0b68056f-openshift-service-ca\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.057880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.057721    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gbnq\" (UniqueName: \"kubernetes.io/projected/7014038a-0215-4164-b032-feff0b68056f-kube-api-access-6gbnq\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.057880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.057810    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-metrics-certs\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.057880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.057850    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.058127 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:49.058098    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-serving-cert: secret "maas-api-e2e-api-a-493ec8-serving-cert" not found
Nov 03 00:25:49.058322 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:49.058185    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls podName:7014038a-0215-4164-b032-feff0b68056f nodeName:}" failed. No retries permitted until 2026-11-03 00:25:49.558163441 +0000 UTC m=+1187.113515458 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls") pod "maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf" (UID: "7014038a-0215-4164-b032-feff0b68056f") : secret "maas-api-e2e-api-a-493ec8-serving-cert" not found
Nov 03 00:25:49.058485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.058463    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7014038a-0215-4164-b032-feff0b68056f-openshift-service-ca\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.060705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.060685    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-metrics-certs\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.080804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.080770    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gbnq\" (UniqueName: \"kubernetes.io/projected/7014038a-0215-4164-b032-feff0b68056f-kube-api-access-6gbnq\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.577977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.577937    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.580884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.580847    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls\") pod \"maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.835102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:49.835019    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:49.911719 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:25:49.910527    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:25:50.171170 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:25:50.171131    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7014038a_0215_4164_b032_feff0b68056f.slice/crio-35a8d181b53c07d5a72923664065a224219d022644d822ab9352e590085dd6a5 WatchSource:0}: Error finding container 35a8d181b53c07d5a72923664065a224219d022644d822ab9352e590085dd6a5: Status 404 returned error can't find the container with id 35a8d181b53c07d5a72923664065a224219d022644d822ab9352e590085dd6a5
Nov 03 00:25:50.171809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:50.171740    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"]
Nov 03 00:25:50.508425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:50.496481    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf" event={"ID":"7014038a-0215-4164-b032-feff0b68056f","Type":"ContainerStarted","Data":"35a8d181b53c07d5a72923664065a224219d022644d822ab9352e590085dd6a5"}
Nov 03 00:25:51.505835 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.505801    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf" event={"ID":"7014038a-0215-4164-b032-feff0b68056f","Type":"ContainerStarted","Data":"e5e822a958f149fab80a7b675a95a4d06ffdfb833d9cfd3448bbc0aa10d165db"}
Nov 03 00:25:51.506322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.505900    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:51.550669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.550638    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"]
Nov 03 00:25:51.555275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.555238    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:51.560193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.560131    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf" podStartSLOduration=3.024669557 podStartE2EDuration="3.560116338s" totalImagesPullingTime="535.446781ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:51.545373571 +0000 UTC m=+1189.100725595" watchObservedRunningTime="2026-11-03 00:25:51.560116338 +0000 UTC m=+1189.115468364"
Nov 03 00:25:51.574720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.574693    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-cb05ae-dockercfg-wzwnd\""
Nov 03 00:25:51.594123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.594093    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-cb05ae\""
Nov 03 00:25:51.594610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.594586    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"]
Nov 03 00:25:51.649913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.649876    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbbvv\" (UniqueName: \"kubernetes.io/projected/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-kube-api-access-dbbvv\") pod \"payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:51.650117 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.649982    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:51.751524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.751492    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbbvv\" (UniqueName: \"kubernetes.io/projected/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-kube-api-access-dbbvv\") pod \"payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:51.751689 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.751588    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:51.752568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.752546    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-plugins-config-volume\") pod \"payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:51.770794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.770720    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbbvv\" (UniqueName: \"kubernetes.io/projected/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-kube-api-access-dbbvv\") pod \"payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") " pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:51.897287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:51.887778    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:52.198593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.198558    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"]
Nov 03 00:25:52.204483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.204455    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.234984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.234949    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"]
Nov 03 00:25:52.297579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.297543    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.297742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.297719    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.297895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.297874    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnx6z\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-kube-api-access-pnx6z\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.298020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.298003    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.298070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.298056    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.298187 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.298163    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.298250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.298233    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.298375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.298346    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.298498 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.298451    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.394499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.394468    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"]
Nov 03 00:25:52.395311 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:25:52.395248    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf516bf3a_d7a5_4653_80f3_f17fb2ed9b42.slice/crio-f855381de3315cb7393ecf01907887ecdf92b2922bf521954481dac50482c2d5 WatchSource:0}: Error finding container f855381de3315cb7393ecf01907887ecdf92b2922bf521954481dac50482c2d5: Status 404 returned error can't find the container with id f855381de3315cb7393ecf01907887ecdf92b2922bf521954481dac50482c2d5
Nov 03 00:25:52.399786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.399718    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.399934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.399822    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnx6z\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-kube-api-access-pnx6z\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.399934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.399917    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.399961    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.399998    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.400038    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.400101    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.400161    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400385 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.400224    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.400480    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istiod-ca-cert\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.400665    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-data\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.400771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.400726    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-certs\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.401649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.401604    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.401649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.401622    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-credential-socket\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.403774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.403750    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-podinfo\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.404022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.403999    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-envoy\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.420083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.420059    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-token\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.421572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.421547    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnx6z\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-kube-api-access-pnx6z\") pod \"e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") " pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.515208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.515167    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w" event={"ID":"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42","Type":"ContainerStarted","Data":"dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3"}
Nov 03 00:25:52.515614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.515239    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w" event={"ID":"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42","Type":"ContainerStarted","Data":"f855381de3315cb7393ecf01907887ecdf92b2922bf521954481dac50482c2d5"}
Nov 03 00:25:52.515614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.515403    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:25:52.530787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.530747    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:52.556534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.556474    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w" podStartSLOduration=1.556454017 podStartE2EDuration="1.556454017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:52.552371816 +0000 UTC m=+1190.107723841" watchObservedRunningTime="2026-11-03 00:25:52.556454017 +0000 UTC m=+1190.111806042"
Nov 03 00:25:52.865399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.865366    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"]
Nov 03 00:25:52.915283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.899851    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:52.915283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.899963    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:52.915283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:52.900001    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:53.580436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:53.578372    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" event={"ID":"e1c1a34a-df81-4149-8b29-7e68e4cd519e","Type":"ContainerStarted","Data":"12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5"}
Nov 03 00:25:53.580436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:53.578453    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" event={"ID":"e1c1a34a-df81-4149-8b29-7e68e4cd519e","Type":"ContainerStarted","Data":"138da4cef30287c63f2822948509f036f6669ef55602f34fc2f0783bdf2aea8c"}
Nov 03 00:25:53.629608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:53.629539    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podStartSLOduration=1.629520248 podStartE2EDuration="1.629520248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:25:53.627800312 +0000 UTC m=+1191.183152348" watchObservedRunningTime="2026-11-03 00:25:53.629520248 +0000 UTC m=+1191.184872272"
Nov 03 00:25:54.531331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.531241    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:25:54.533248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.533221    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 00:25:54.533394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.533316    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 00:25:54.573525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.573495    6000 generic.go:356] "Generic (PLEG): container finished" podID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerID="daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135" exitCode=0
Nov 03 00:25:54.573677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.573558    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" event={"ID":"b8b6fc11-a279-4997-a8bf-c196b8650d8c","Type":"ContainerDied","Data":"daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135"}
Nov 03 00:25:54.573989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.573972    6000 scope.go:122] "RemoveContainer" containerID="daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135"
Nov 03 00:25:54.575465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.575436    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:54.575555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.575507    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:54.575555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:54.575544    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:25:55.546282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:55.542679    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 00:25:55.546282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:55.542735    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 00:25:55.633797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:55.633764    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" event={"ID":"b8b6fc11-a279-4997-a8bf-c196b8650d8c","Type":"ContainerStarted","Data":"9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680"}
Nov 03 00:25:55.983268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:55.983233    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:25:55.983676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:55.983647    6000 patch_prober.go:29] interesting pod/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:25:55.983784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:55.983712    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 00:25:56.540284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:56.532491    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 00:25:56.540284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:56.532556    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 00:25:56.983550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:56.983515    6000 patch_prober.go:29] interesting pod/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:25:56.984010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:56.983578    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 00:25:57.162919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:57.162882    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"]
Nov 03 00:25:57.163295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:57.163249    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5" podUID="6faf3a50-faf8-4b84-a99f-6f59f8b6140b" containerName="maas-api" containerID="cri-o://109d40da868b8b45c7e2a84693d46bfbb0a2ad2f384303beceefdb0011813633" gracePeriod=30
Nov 03 00:25:57.568337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:57.523134    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:25:57.793341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:57.788963    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 00:25:57.793341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:57.789042    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 00:25:58.316674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.084001    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:25:58.316674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.147358    6000 patch_prober.go:29] interesting pod/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:25:58.316674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.147451    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 00:25:58.742529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.733602    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 00:25:58.866191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.864145    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-m2dbv" podUID="f6adde23-6bf2-4a92-8002-a6e093c1c2ba" containerName="main" probeResult="failure" output="Get \"https://10.134.0.36:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:58.866191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.865173    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:58.866191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.865559    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": context deadline exceeded"
Nov 03 00:25:58.867646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.867541    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:25:58.874295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.874270    6000 generic.go:356] "Generic (PLEG): container finished" podID="6faf3a50-faf8-4b84-a99f-6f59f8b6140b" containerID="109d40da868b8b45c7e2a84693d46bfbb0a2ad2f384303beceefdb0011813633" exitCode=0
Nov 03 00:25:58.881722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.881698    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5" event={"ID":"6faf3a50-faf8-4b84-a99f-6f59f8b6140b","Type":"ContainerDied","Data":"109d40da868b8b45c7e2a84693d46bfbb0a2ad2f384303beceefdb0011813633"}
Nov 03 00:25:58.883129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.882813    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 00:25:58.890426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.890404    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:25:58.983601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.983562    6000 patch_prober.go:29] interesting pod/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:25:58.983750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:58.983616    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 00:25:59.013742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.013704    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-metrics-certs\") pod \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") "
Nov 03 00:25:59.014034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.014014    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-kube-api-access-nv769\" (UniqueName: \"kubernetes.io/projected/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-kube-api-access-nv769\") pod \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") "
Nov 03 00:25:59.014338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.014319    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-maas-api-tls\") pod \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") "
Nov 03 00:25:59.014671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.014653    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-openshift-service-ca\") pod \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\" (UID: \"6faf3a50-faf8-4b84-a99f-6f59f8b6140b\") "
Nov 03 00:25:59.051280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.051227    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-openshift-service-ca" pod "6faf3a50-faf8-4b84-a99f-6f59f8b6140b" (UID: "6faf3a50-faf8-4b84-a99f-6f59f8b6140b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:25:59.053731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.053706    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-kube-api-access-nv769" pod "6faf3a50-faf8-4b84-a99f-6f59f8b6140b" (UID: "6faf3a50-faf8-4b84-a99f-6f59f8b6140b"). InnerVolumeSpecName "kube-api-access-nv769". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:25:59.054335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.054314    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-metrics-certs" pod "6faf3a50-faf8-4b84-a99f-6f59f8b6140b" (UID: "6faf3a50-faf8-4b84-a99f-6f59f8b6140b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:59.054691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.054665    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-maas-api-tls" pod "6faf3a50-faf8-4b84-a99f-6f59f8b6140b" (UID: "6faf3a50-faf8-4b84-a99f-6f59f8b6140b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:25:59.120812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.120765    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:59.141708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.141618    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:59.142070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.142049    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nv769\" (UniqueName: \"kubernetes.io/projected/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-kube-api-access-nv769\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:59.142188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.142176    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6faf3a50-faf8-4b84-a99f-6f59f8b6140b-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:25:59.533340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.532677    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 00:25:59.533340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.532737    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 00:25:59.908945 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.908879    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5" event={"ID":"6faf3a50-faf8-4b84-a99f-6f59f8b6140b","Type":"ContainerDied","Data":"9c2413cc2d8c3632f0ff09a8ca8c419c3c3c5119c64fc9d4ff4e6b1d0c57d64c"}
Nov 03 00:25:59.909235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.909215    6000 scope.go:122] "RemoveContainer" containerID="109d40da868b8b45c7e2a84693d46bfbb0a2ad2f384303beceefdb0011813633"
Nov 03 00:25:59.909549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.909532    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"
Nov 03 00:25:59.982792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.982753    6000 patch_prober.go:29] interesting pod/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:25:59.983004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:25:59.982832    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 00:26:00.030341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:00.030304    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"]
Nov 03 00:26:00.042986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:00.042948    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-4aa4e9-8585d96bc6-jgkx5"]
Nov 03 00:26:00.532959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:00.532917    6000 patch_prober.go:29] interesting pod/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 03 00:26:00.533282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:00.533221    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 03 00:26:00.572364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:00.571506    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6faf3a50-faf8-4b84-a99f-6f59f8b6140b" path="/var/lib/kubelet/pods/6faf3a50-faf8-4b84-a99f-6f59f8b6140b/volumes"
Nov 03 00:26:00.982850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:00.982691    6000 patch_prober.go:29] interesting pod/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:26:00.982850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:00.982787    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 00:26:01.606553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:01.606365    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"]
Nov 03 00:26:01.625034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:01.608375    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy" containerID="cri-o://cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b" gracePeriod=30
Nov 03 00:26:01.733861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:01.732426    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"]
Nov 03 00:26:01.733861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:01.732666    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy" containerID="cri-o://12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5" gracePeriod=30
Nov 03 00:26:01.753550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:01.741534    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:26:01.974900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:01.974784    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"]
Nov 03 00:26:01.975606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:01.975552    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy" containerID="cri-o://6085d7b3bdc8f09b0ba9ba9354e07f0343e48379148a70e6ff0817fd88f08759" gracePeriod=30
Nov 03 00:26:02.023102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:02.023059    6000 patch_prober.go:29] interesting pod/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 03 00:26:02.048598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:02.048548    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 03 00:26:02.338668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:02.338572    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:26:02.959997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:02.959962    6000 scope.go:122] "RemoveContainer" containerID="993c6193fff99e3c3789989388b35856217d1c7cc098c99752234a7690dd8d94"
Nov 03 00:26:03.013381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:02.997790    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:26:03.120521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.120488    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"]
Nov 03 00:26:03.121993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.121614    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6faf3a50-faf8-4b84-a99f-6f59f8b6140b" containerName="maas-api"
Nov 03 00:26:03.121993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.121638    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6faf3a50-faf8-4b84-a99f-6f59f8b6140b" containerName="maas-api"
Nov 03 00:26:03.121993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.121822    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6faf3a50-faf8-4b84-a99f-6f59f8b6140b" containerName="maas-api"
Nov 03 00:26:03.128250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.127942    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"]
Nov 03 00:26:03.128250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.128088    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.142121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.142094    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-3437aa-openshift-default-dockercfg-5prrt\""
Nov 03 00:26:03.170942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.170905    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-socket\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.170950    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-data\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.171037    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-token\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.171088    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istiod-ca-cert\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.171108    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-envoy\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.171142    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-kube-api-access-kfntz\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.171164    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-credential-socket\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.171200    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-podinfo\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.171452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.171242    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-certs\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.222635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.222560    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285449    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-token\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285575    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istiod-ca-cert\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285613    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-envoy\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285682    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-kube-api-access-kfntz\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285717    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-credential-socket\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285778    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-podinfo\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285844    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-certs\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.285986    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-socket\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.286021    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-data\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.296316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.286571    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-data\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.308962 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.304619    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istiod-ca-cert\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.325315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.316024    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-certs\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.348517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.336589    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-socket\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.348517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.338858    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-credential-socket\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.394281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.394211    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-podinfo\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.407293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.394708    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-token\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.644293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.641684    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-kube-api-access-kfntz\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.682278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.680008    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:26:03.682491 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.682352    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-envoy\") pod \"e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") " pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:03.744678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:03.744450    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:04.116823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:04.116610    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"]
Nov 03 00:26:04.120394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:04.120362    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:26:04.120468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:04.120426    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:26:04.120468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:04.120455    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:26:04.255591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:04.255400    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" event={"ID":"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb","Type":"ContainerStarted","Data":"2ed1a73e022b74f0426583f5f455be85e6d5d04b557a7b2903e19635280494e6"}
Nov 03 00:26:05.262068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:05.262028    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" event={"ID":"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb","Type":"ContainerStarted","Data":"86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3"}
Nov 03 00:26:05.340299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:05.340230    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podStartSLOduration=2.340215329 podStartE2EDuration="2.340215329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:05.337715209 +0000 UTC m=+1202.893067233" watchObservedRunningTime="2026-11-03 00:26:05.340215329 +0000 UTC m=+1202.895567355"
Nov 03 00:26:05.745487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:05.745452    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:05.750877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:05.750854    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:06.269215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:06.269187    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:06.270418 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:06.270396    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:26:07.077657 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:07.076437    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19317f67_59c8_4496_9e87_734482cca794.slice/crio-cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c1a34a_df81_4149_8b29_7e68e4cd519e.slice/crio-conmon-12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1c1a34a_df81_4149_8b29_7e68e4cd519e.slice/crio-12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40eafd36_9dc8_4649_9b0f_b4d13ade4390.slice/crio-conmon-6085d7b3bdc8f09b0ba9ba9354e07f0343e48379148a70e6ff0817fd88f08759.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:26:07.207231 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.207150    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:26:07.223874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.223845    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252357    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-socket\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252427    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-credential-socket\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252457    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-credential-socket\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252561    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-certs\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252616    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istiod-ca-cert\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252659    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-socket" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252701    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19317f67-59c8-4496-9e87-734482cca794-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19317f67-59c8-4496-9e87-734482cca794-istiod-ca-cert\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252724    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-credential-socket" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252734    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-token\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-token\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252814    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-envoy\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252856    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-data\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252884    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-podinfo\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252909    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-certs" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252937    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-kube-api-access-pnx6z\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-kube-api-access-pnx6z\") pod \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\" (UID: \"e1c1a34a-df81-4149-8b29-7e68e4cd519e\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.252993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-kube-api-access-xjdkr\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-kube-api-access-xjdkr\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253025    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-certs\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253076    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19317f67-59c8-4496-9e87-734482cca794-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19317f67-59c8-4496-9e87-734482cca794-istio-podinfo\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253108    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-credential-socket" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253158    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-istio-token\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-istio-token\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253190    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-envoy\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253220    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-data\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253274    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-socket\") pod \"19317f67-59c8-4496-9e87-734482cca794\" (UID: \"19317f67-59c8-4496-9e87-734482cca794\") "
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253275    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istiod-ca-cert" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.253504    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-socket" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254042    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254063    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254076    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254090    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254103    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254116    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254433    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-data" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.254779    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19317f67-59c8-4496-9e87-734482cca794-istiod-ca-cert" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.255095    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-data" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.255116    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-certs" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.257642    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-kube-api-access-xjdkr" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "kube-api-access-xjdkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.258803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.258541    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-envoy" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.268274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.264649    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-envoy" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.268274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.264807    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-istio-token" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.268274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.265738    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19317f67-59c8-4496-9e87-734482cca794-istio-podinfo" pod "19317f67-59c8-4496-9e87-734482cca794" (UID: "19317f67-59c8-4496-9e87-734482cca794"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:26:07.268274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.265757    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-podinfo" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:26:07.268274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.266442    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-kube-api-access-pnx6z" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "kube-api-access-pnx6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.282633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.273743    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-token" pod "e1c1a34a-df81-4149-8b29-7e68e4cd519e" (UID: "e1c1a34a-df81-4149-8b29-7e68e4cd519e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.310694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.310235    6000 generic.go:356] "Generic (PLEG): container finished" podID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerID="12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5" exitCode=0
Nov 03 00:26:07.310694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.310462    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" event={"ID":"e1c1a34a-df81-4149-8b29-7e68e4cd519e","Type":"ContainerDied","Data":"12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5"}
Nov 03 00:26:07.310694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.310551    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk" event={"ID":"e1c1a34a-df81-4149-8b29-7e68e4cd519e","Type":"ContainerDied","Data":"138da4cef30287c63f2822948509f036f6669ef55602f34fc2f0783bdf2aea8c"}
Nov 03 00:26:07.310694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.310584    6000 scope.go:122] "RemoveContainer" containerID="12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5"
Nov 03 00:26:07.311056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.310730    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"
Nov 03 00:26:07.323442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.323396    6000 generic.go:356] "Generic (PLEG): container finished" podID="19317f67-59c8-4496-9e87-734482cca794" containerID="cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b" exitCode=0
Nov 03 00:26:07.323790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.323551    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"
Nov 03 00:26:07.324551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.323551    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" event={"ID":"19317f67-59c8-4496-9e87-734482cca794","Type":"ContainerDied","Data":"cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b"}
Nov 03 00:26:07.324551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.324448    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc" event={"ID":"19317f67-59c8-4496-9e87-734482cca794","Type":"ContainerDied","Data":"b277089eaeef0f1da84fb28ebb2635494bb4b2144db749822457f9b373ef6b2b"}
Nov 03 00:26:07.327204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.327181    6000 generic.go:356] "Generic (PLEG): container finished" podID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerID="6085d7b3bdc8f09b0ba9ba9354e07f0343e48379148a70e6ff0817fd88f08759" exitCode=0
Nov 03 00:26:07.327373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.327348    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" event={"ID":"40eafd36-9dc8-4649-9b0f-b4d13ade4390","Type":"ContainerDied","Data":"6085d7b3bdc8f09b0ba9ba9354e07f0343e48379148a70e6ff0817fd88f08759"}
Nov 03 00:26:07.332249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.332151    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:26:07.343406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.343386    6000 scope.go:122] "RemoveContainer" containerID="12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5"
Nov 03 00:26:07.344138 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:07.344102    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5\": container with ID starting with 12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5 not found: ID does not exist" containerID="12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5"
Nov 03 00:26:07.344245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.344140    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5"} err="failed to get container status \"12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5\": rpc error: code = NotFound desc = could not find container \"12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5\": container with ID starting with 12e3dd102896a2124a5ee2126327ea28aeccd7eb202af801676daa17db28b9f5 not found: ID does not exist"
Nov 03 00:26:07.344245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.344161    6000 scope.go:122] "RemoveContainer" containerID="cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b"
Nov 03 00:26:07.356723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.356697    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-socket\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.357095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357075    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-token\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-token\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.357296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357280    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-podinfo\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.357412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357399    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istiod-ca-cert\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.357521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357509    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-envoy\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.357737 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357720    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-credential-socket\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.357886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357860    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-kube-api-access-mbp4z\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-kube-api-access-mbp4z\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.357992 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357953    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-certs\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.358047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.358013    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-data\") pod \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\" (UID: \"40eafd36-9dc8-4649-9b0f-b4d13ade4390\") "
Nov 03 00:26:07.359154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359129    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19317f67-59c8-4496-9e87-734482cca794-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359304    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359424    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359512    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359595    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e1c1a34a-df81-4149-8b29-7e68e4cd519e-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359677    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnx6z\" (UniqueName: \"kubernetes.io/projected/e1c1a34a-df81-4149-8b29-7e68e4cd519e-kube-api-access-pnx6z\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359781 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359768    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjdkr\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-kube-api-access-xjdkr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359851    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.359942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.359932    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19317f67-59c8-4496-9e87-734482cca794-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.360025 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.360015    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19317f67-59c8-4496-9e87-734482cca794-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.360102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.360092    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.360186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.360176    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19317f67-59c8-4496-9e87-734482cca794-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.360671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.360648    6000 scope.go:122] "RemoveContainer" containerID="cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b"
Nov 03 00:26:07.361809 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:07.361785    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b\": container with ID starting with cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b not found: ID does not exist" containerID="cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b"
Nov 03 00:26:07.361907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.361818    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b"} err="failed to get container status \"cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b\": rpc error: code = NotFound desc = could not find container \"cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b\": container with ID starting with cae1ff6b62c5e2266f7f7a71a978991a03a5e028b1b5d1279458d762990d119b not found: ID does not exist"
Nov 03 00:26:07.361907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.361840    6000 scope.go:122] "RemoveContainer" containerID="b04201a0975b3b0ed651aa1f0ce72907cbb4e785046ce0fa15656033e1917e80"
Nov 03 00:26:07.362520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.357543    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-socket" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.362618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.360554    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-token" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.362618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.360910    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-certs" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.362618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.360992    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istiod-ca-cert" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:07.362618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.361214    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-data" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.362618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.362497    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-credential-socket" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.363023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.362998    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-podinfo" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:26:07.365433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.365411    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-kube-api-access-mbp4z" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "kube-api-access-mbp4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:07.367351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.367237    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-envoy" pod "40eafd36-9dc8-4649-9b0f-b4d13ade4390" (UID: "40eafd36-9dc8-4649-9b0f-b4d13ade4390"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:26:07.383472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.383443    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"]
Nov 03 00:26:07.389984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.389958    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-w5blk"]
Nov 03 00:26:07.460563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.460407    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"]
Nov 03 00:26:07.461873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461844    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461877    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461891    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461906    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461918    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461933    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbp4z\" (UniqueName: \"kubernetes.io/projected/40eafd36-9dc8-4649-9b0f-b4d13ade4390-kube-api-access-mbp4z\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461948    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461961    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.462002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.461973    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40eafd36-9dc8-4649-9b0f-b4d13ade4390-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:07.465742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:07.465715    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-pwdtc"]
Nov 03 00:26:08.338770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.338679    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2" event={"ID":"40eafd36-9dc8-4649-9b0f-b4d13ade4390","Type":"ContainerDied","Data":"248db4723c622cc50f0a76b422add99f5ac9f537f1acfbef2a52feec93df2576"}
Nov 03 00:26:08.338770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.338760    6000 scope.go:122] "RemoveContainer" containerID="6085d7b3bdc8f09b0ba9ba9354e07f0343e48379148a70e6ff0817fd88f08759"
Nov 03 00:26:08.339373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.338864    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"
Nov 03 00:26:08.432183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.432147    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"]
Nov 03 00:26:08.437318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.437287    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-4aa4e9-openshift-default-8576d9c975-zmrb2"]
Nov 03 00:26:08.569698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.569668    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19317f67-59c8-4496-9e87-734482cca794" path="/var/lib/kubelet/pods/19317f67-59c8-4496-9e87-734482cca794/volumes"
Nov 03 00:26:08.570187 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.570174    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" path="/var/lib/kubelet/pods/40eafd36-9dc8-4649-9b0f-b4d13ade4390/volumes"
Nov 03 00:26:08.570692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:08.570678    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" path="/var/lib/kubelet/pods/e1c1a34a-df81-4149-8b29-7e68e4cd519e/volumes"
Nov 03 00:26:09.874448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.874416    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"]
Nov 03 00:26:09.918286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.914112    6000 cpu_manager.go:414] "containerMap: removing container" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy"
Nov 03 00:26:09.918497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918317    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy"
Nov 03 00:26:09.918497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918410    6000 cpu_manager.go:414] "containerMap: removing container" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy"
Nov 03 00:26:09.918497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918421    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy"
Nov 03 00:26:09.918497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918492    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy"
Nov 03 00:26:09.918720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918499    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy"
Nov 03 00:26:09.918778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918742    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy"
Nov 03 00:26:09.918778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918765    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy"
Nov 03 00:26:09.918878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918784    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="19317f67-59c8-4496-9e87-734482cca794" containerName="istio-proxy"
Nov 03 00:26:09.918878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:09.918803    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1c1a34a-df81-4149-8b29-7e68e4cd519e" containerName="istio-proxy"
Nov 03 00:26:10.006276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.005144    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"]
Nov 03 00:26:10.006276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.005303    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.062583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.054683    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-cb05ae-serving-cert\""
Nov 03 00:26:10.164371 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.164285    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-maas-api-tls\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.164530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.164417    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4609dd8-273d-43a9-94ac-70e51d339fa2-openshift-service-ca\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.164621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.164597    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7v7k\" (UniqueName: \"kubernetes.io/projected/f4609dd8-273d-43a9-94ac-70e51d339fa2-kube-api-access-v7v7k\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.164675 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.164664    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-metrics-certs\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.268276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.267479    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-metrics-certs\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.268276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.267814    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-maas-api-tls\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.268276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.267893    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4609dd8-273d-43a9-94ac-70e51d339fa2-openshift-service-ca\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.268276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.268163    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7v7k\" (UniqueName: \"kubernetes.io/projected/f4609dd8-273d-43a9-94ac-70e51d339fa2-kube-api-access-v7v7k\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.281303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.272916    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4609dd8-273d-43a9-94ac-70e51d339fa2-openshift-service-ca\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.301277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.298463    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-metrics-certs\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.316279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.309324    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-maas-api-tls\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.316279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.314765    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7v7k\" (UniqueName: \"kubernetes.io/projected/f4609dd8-273d-43a9-94ac-70e51d339fa2-kube-api-access-v7v7k\") pod \"maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.388475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.388444    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:10.698657 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:26:10.698592    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4609dd8_273d_43a9_94ac_70e51d339fa2.slice/crio-f280440f59d0a81332106d4a9e1c89beaeb13203237e6859c19ce8e6738916b0 WatchSource:0}: Error finding container f280440f59d0a81332106d4a9e1c89beaeb13203237e6859c19ce8e6738916b0: Status 404 returned error can't find the container with id f280440f59d0a81332106d4a9e1c89beaeb13203237e6859c19ce8e6738916b0
Nov 03 00:26:10.699049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:10.699026    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"]
Nov 03 00:26:11.413933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:11.413902    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz" event={"ID":"f4609dd8-273d-43a9-94ac-70e51d339fa2","Type":"ContainerStarted","Data":"2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47"}
Nov 03 00:26:11.415497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:11.413953    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz" event={"ID":"f4609dd8-273d-43a9-94ac-70e51d339fa2","Type":"ContainerStarted","Data":"f280440f59d0a81332106d4a9e1c89beaeb13203237e6859c19ce8e6738916b0"}
Nov 03 00:26:11.415497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:11.414026    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:11.454849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:11.454786    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz" podStartSLOduration=2.103880451 podStartE2EDuration="2.454768045s" totalImagesPullingTime="350.887594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:11.452199212 +0000 UTC m=+1209.007551235" watchObservedRunningTime="2026-11-03 00:26:11.454768045 +0000 UTC m=+1209.010120070"
Nov 03 00:26:12.423595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:12.423564    6000 generic.go:356] "Generic (PLEG): container finished" podID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerID="8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b" exitCode=0
Nov 03 00:26:12.424063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:12.423635    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" event={"ID":"93aecdc9-6974-42f6-b72d-c6da546c6c91","Type":"ContainerDied","Data":"8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b"}
Nov 03 00:26:12.424139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:12.424062    6000 scope.go:122] "RemoveContainer" containerID="8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b"
Nov 03 00:26:12.425512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:12.425484    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:26:12.425637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:12.425564    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:26:12.425637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:12.425612    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:26:13.445600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:13.445561    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" event={"ID":"93aecdc9-6974-42f6-b72d-c6da546c6c91","Type":"ContainerStarted","Data":"c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810"}
Nov 03 00:26:13.636739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:13.636706    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:26:14.326283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:14.325603    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:26:14.331281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:14.328395    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:26:14.331281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:14.328458    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 00:26:15.323676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:15.323636    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:26:15.324119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:15.323698    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 00:26:16.327282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:16.325843    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:26:16.327282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:16.325919    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 00:26:17.323767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:17.323733    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:26:17.323949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:17.323797    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 00:26:17.429169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:17.429138    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:26:18.096338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:17.984388    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"]
Nov 03 00:26:18.096338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:17.993384    6000 cpu_manager.go:414] "containerMap: removing container" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy"
Nov 03 00:26:18.096338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:17.993404    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40eafd36-9dc8-4649-9b0f-b4d13ade4390" containerName="istio-proxy"
Nov 03 00:26:18.149519 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.149477    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"]
Nov 03 00:26:18.149718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.149621    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.292284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.272760    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-27490b-serving-cert\""
Nov 03 00:26:18.292567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.284676    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-maas-api-tls\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.292810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.292786    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44e2389-13b8-4509-89e0-409888b5edb8-openshift-service-ca\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.293138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.293113    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-metrics-certs\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.293425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.293395    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/a44e2389-13b8-4509-89e0-409888b5edb8-kube-api-access-qnr5n\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.448278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.443499    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-maas-api-tls\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.448278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.443670    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44e2389-13b8-4509-89e0-409888b5edb8-openshift-service-ca\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.448278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.443962    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-metrics-certs\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.448278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.444129    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/a44e2389-13b8-4509-89e0-409888b5edb8-kube-api-access-qnr5n\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.448278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.445327    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 03 00:26:18.448278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.447687    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44e2389-13b8-4509-89e0-409888b5edb8-openshift-service-ca\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.448278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.447810    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 03 00:26:18.507615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.502322    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-metrics-certs\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.507615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.504001    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-maas-api-tls\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.513846 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.513772    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/a44e2389-13b8-4509-89e0-409888b5edb8-kube-api-access-qnr5n\") pod \"maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:18.681313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:18.681282    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:19.010416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:19.010384    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"]
Nov 03 00:26:19.010650 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:26:19.010625    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44e2389_13b8_4509_89e0_409888b5edb8.slice/crio-0d892919f2f67f77e26927cd60c5797c6297ee0f68bcccd447ce02e93e8a91d6 WatchSource:0}: Error finding container 0d892919f2f67f77e26927cd60c5797c6297ee0f68bcccd447ce02e93e8a91d6: Status 404 returned error can't find the container with id 0d892919f2f67f77e26927cd60c5797c6297ee0f68bcccd447ce02e93e8a91d6
Nov 03 00:26:19.368635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:19.368577    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:26:19.371338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:19.371311    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:26:19.611413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:19.611324    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v" event={"ID":"a44e2389-13b8-4509-89e0-409888b5edb8","Type":"ContainerStarted","Data":"0d892919f2f67f77e26927cd60c5797c6297ee0f68bcccd447ce02e93e8a91d6"}
Nov 03 00:26:19.898755 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:19.898664    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:26:20.620111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:20.620075    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v" event={"ID":"a44e2389-13b8-4509-89e0-409888b5edb8","Type":"ContainerStarted","Data":"a1346a3c27899c9a7f87d6856047071ed931108832eb7b84ffd42f0dec01603d"}
Nov 03 00:26:20.620586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:20.620162    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:20.656885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:20.656831    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v" podStartSLOduration=3.307787326 podStartE2EDuration="3.656813975s" totalImagesPullingTime="349.026649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:20.656597861 +0000 UTC m=+1218.211949901" watchObservedRunningTime="2026-11-03 00:26:20.656813975 +0000 UTC m=+1218.212166012"
Nov 03 00:26:23.572911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.572879    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"]
Nov 03 00:26:23.576455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.576437    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:23.596220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.595497    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"]
Nov 03 00:26:23.602743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.602714    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d43409-dockercfg-pdj7r\""
Nov 03 00:26:23.602743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.602726    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d43409\""
Nov 03 00:26:23.772628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.768818    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0de994-cdee-4f27-b8ca-860b8e31b169-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:23.772628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.769174    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6cbd\" (UniqueName: \"kubernetes.io/projected/aa0de994-cdee-4f27-b8ca-860b8e31b169-kube-api-access-s6cbd\") pod \"payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:23.943316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.938840    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6cbd\" (UniqueName: \"kubernetes.io/projected/aa0de994-cdee-4f27-b8ca-860b8e31b169-kube-api-access-s6cbd\") pod \"payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:23.943316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.939104    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0de994-cdee-4f27-b8ca-860b8e31b169-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:23.943316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.940233    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0de994-cdee-4f27-b8ca-860b8e31b169-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:23.991602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:23.991564    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6cbd\" (UniqueName: \"kubernetes.io/projected/aa0de994-cdee-4f27-b8ca-860b8e31b169-kube-api-access-s6cbd\") pod \"payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:24.190928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:24.190890    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:24.489993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:24.489963    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"]
Nov 03 00:26:24.490243 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:26:24.490220    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa0de994_cdee_4f27_b8ca_860b8e31b169.slice/crio-279e1aad48dfa130cd580571db55ddaeb928926590f2756c2ef114add986c140 WatchSource:0}: Error finding container 279e1aad48dfa130cd580571db55ddaeb928926590f2756c2ef114add986c140: Status 404 returned error can't find the container with id 279e1aad48dfa130cd580571db55ddaeb928926590f2756c2ef114add986c140
Nov 03 00:26:25.014460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:25.014418    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8" event={"ID":"aa0de994-cdee-4f27-b8ca-860b8e31b169","Type":"ContainerStarted","Data":"add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4"}
Nov 03 00:26:25.014930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:25.014524    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8" event={"ID":"aa0de994-cdee-4f27-b8ca-860b8e31b169","Type":"ContainerStarted","Data":"279e1aad48dfa130cd580571db55ddaeb928926590f2756c2ef114add986c140"}
Nov 03 00:26:25.014930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:25.014554    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:25.052333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:25.052270    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8" podStartSLOduration=2.052238821 podStartE2EDuration="2.052238821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:25.050426381 +0000 UTC m=+1222.605778428" watchObservedRunningTime="2026-11-03 00:26:25.052238821 +0000 UTC m=+1222.607590856"
Nov 03 00:26:26.633789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:26.633759    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:26:30.082703 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.082662    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"]
Nov 03 00:26:30.086228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.086211    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.113399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.113370    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-3437aa-serving-cert\""
Nov 03 00:26:30.118683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.118656    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"]
Nov 03 00:26:30.194611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.194581    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h89t2\" (UniqueName: \"kubernetes.io/projected/c1c25b44-8d36-473c-9878-fbafc221423c-kube-api-access-h89t2\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.194772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.194736    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.194840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.194827    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c25b44-8d36-473c-9878-fbafc221423c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.194880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.194872    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-metrics-certs\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.300456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.300386    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h89t2\" (UniqueName: \"kubernetes.io/projected/c1c25b44-8d36-473c-9878-fbafc221423c-kube-api-access-h89t2\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.300857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.300836    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.301035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.301019    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c25b44-8d36-473c-9878-fbafc221423c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.301202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.301187    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-metrics-certs\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.302031 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:30.302009    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-serving-cert: secret "maas-api-e2e-ipp-a-3437aa-serving-cert" not found
Nov 03 00:26:30.303716 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:30.303626    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls podName:c1c25b44-8d36-473c-9878-fbafc221423c nodeName:}" failed. No retries permitted until 2026-11-03 00:26:30.803605342 +0000 UTC m=+1228.358957350 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls") pod "maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6" (UID: "c1c25b44-8d36-473c-9878-fbafc221423c") : secret "maas-api-e2e-ipp-a-3437aa-serving-cert" not found
Nov 03 00:26:30.304159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.303506    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c25b44-8d36-473c-9878-fbafc221423c-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.307648 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.307600    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-metrics-certs\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.331734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.331661    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h89t2\" (UniqueName: \"kubernetes.io/projected/c1c25b44-8d36-473c-9878-fbafc221423c-kube-api-access-h89t2\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.807826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.807794    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.810377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.810354    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls\") pod \"maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:30.998962 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:30.998931    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:31.297766 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:26:31.297739    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c25b44_8d36_473c_9878_fbafc221423c.slice/crio-923fd1ac6f88cb8cac9572db73956accad5bd2be1512ea4e8ffd22b967be6272 WatchSource:0}: Error finding container 923fd1ac6f88cb8cac9572db73956accad5bd2be1512ea4e8ffd22b967be6272: Status 404 returned error can't find the container with id 923fd1ac6f88cb8cac9572db73956accad5bd2be1512ea4e8ffd22b967be6272
Nov 03 00:26:31.298944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:31.298917    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"]
Nov 03 00:26:32.103551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:32.103513    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6" event={"ID":"c1c25b44-8d36-473c-9878-fbafc221423c","Type":"ContainerStarted","Data":"1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58"}
Nov 03 00:26:32.103709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:32.103582    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6" event={"ID":"c1c25b44-8d36-473c-9878-fbafc221423c","Type":"ContainerStarted","Data":"923fd1ac6f88cb8cac9572db73956accad5bd2be1512ea4e8ffd22b967be6272"}
Nov 03 00:26:32.103783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:32.103769    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:32.141614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:32.141564    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6" podStartSLOduration=1.8350297580000001 podStartE2EDuration="2.141549634s" totalImagesPullingTime="306.519876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:32.1393799 +0000 UTC m=+1229.694731923" watchObservedRunningTime="2026-11-03 00:26:32.141549634 +0000 UTC m=+1229.696901685"
Nov 03 00:26:33.388282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:33.381620    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:26:36.025366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:36.025338    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:26:38.117734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:38.117699    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:26:46.371753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.371718    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"]
Nov 03 00:26:46.377190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.377168    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.401617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.401586    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"]
Nov 03 00:26:46.407635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.407609    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-d43409-serving-cert\""
Nov 03 00:26:46.435557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.435522    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-metrics-certs\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.435751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.435734    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-maas-api-tls\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.435866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.435849    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85f06b-2d34-4260-bba7-1a86a0135c9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.435974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.435955    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2z8d\" (UniqueName: \"kubernetes.io/projected/8c85f06b-2d34-4260-bba7-1a86a0135c9b-kube-api-access-r2z8d\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.537291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.537241    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-maas-api-tls\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.537493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.537367    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85f06b-2d34-4260-bba7-1a86a0135c9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.537493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.537476    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2z8d\" (UniqueName: \"kubernetes.io/projected/8c85f06b-2d34-4260-bba7-1a86a0135c9b-kube-api-access-r2z8d\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.537640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.537626    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-metrics-certs\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.538057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.537988    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85f06b-2d34-4260-bba7-1a86a0135c9b-openshift-service-ca\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.540016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.539994    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-maas-api-tls\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.540201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.540178    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-metrics-certs\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.556760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.556725    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2z8d\" (UniqueName: \"kubernetes.io/projected/8c85f06b-2d34-4260-bba7-1a86a0135c9b-kube-api-access-r2z8d\") pod \"maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:46.693829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:46.693743    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:47.216987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:47.216893    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"]
Nov 03 00:26:47.243663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:47.243634    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr" event={"ID":"8c85f06b-2d34-4260-bba7-1a86a0135c9b","Type":"ContainerStarted","Data":"abd365c9c3a86d3205e46b1e20d9e4310e23974d4a32ba2974264092ba29e4ac"}
Nov 03 00:26:48.263538 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:48.263498    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr" event={"ID":"8c85f06b-2d34-4260-bba7-1a86a0135c9b","Type":"ContainerStarted","Data":"1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9"}
Nov 03 00:26:48.265655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:48.265624    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:49.897046 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:49.897011    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:26:50.433282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:50.420815    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr" podStartSLOduration=4.002363305 podStartE2EDuration="4.420788034s" totalImagesPullingTime="418.424729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:26:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:26:48.35354285 +0000 UTC m=+1245.908894902" watchObservedRunningTime="2026-11-03 00:26:50.420788034 +0000 UTC m=+1247.976140062"
Nov 03 00:26:50.452279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:50.436982    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"]
Nov 03 00:26:50.453279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:50.453224    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6" podUID="403dedd4-49e9-43de-a391-d0da930a91fe" containerName="maas-api" containerID="cri-o://fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad" gracePeriod=30
Nov 03 00:26:50.960318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:50.960234    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:26:51.086282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.085238    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls\") pod \"403dedd4-49e9-43de-a391-d0da930a91fe\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") "
Nov 03 00:26:51.086282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.085450    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403dedd4-49e9-43de-a391-d0da930a91fe-kube-api-access-lsnq8\" (UniqueName: \"kubernetes.io/projected/403dedd4-49e9-43de-a391-d0da930a91fe-kube-api-access-lsnq8\") pod \"403dedd4-49e9-43de-a391-d0da930a91fe\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") "
Nov 03 00:26:51.086282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.085682    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403dedd4-49e9-43de-a391-d0da930a91fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403dedd4-49e9-43de-a391-d0da930a91fe-openshift-service-ca\") pod \"403dedd4-49e9-43de-a391-d0da930a91fe\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") "
Nov 03 00:26:51.086282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.085717    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-metrics-certs\") pod \"403dedd4-49e9-43de-a391-d0da930a91fe\" (UID: \"403dedd4-49e9-43de-a391-d0da930a91fe\") "
Nov 03 00:26:51.114277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.107209    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-metrics-certs" pod "403dedd4-49e9-43de-a391-d0da930a91fe" (UID: "403dedd4-49e9-43de-a391-d0da930a91fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:26:51.145473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.134586    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403dedd4-49e9-43de-a391-d0da930a91fe-openshift-service-ca" pod "403dedd4-49e9-43de-a391-d0da930a91fe" (UID: "403dedd4-49e9-43de-a391-d0da930a91fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:26:51.145473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.141595    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls" pod "403dedd4-49e9-43de-a391-d0da930a91fe" (UID: "403dedd4-49e9-43de-a391-d0da930a91fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:26:51.216247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.216206    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403dedd4-49e9-43de-a391-d0da930a91fe-kube-api-access-lsnq8" pod "403dedd4-49e9-43de-a391-d0da930a91fe" (UID: "403dedd4-49e9-43de-a391-d0da930a91fe"). InnerVolumeSpecName "kube-api-access-lsnq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:26:51.224392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.220428    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/403dedd4-49e9-43de-a391-d0da930a91fe-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:51.224392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.220458    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:51.224392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.220472    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/403dedd4-49e9-43de-a391-d0da930a91fe-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:51.224392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.220485    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsnq8\" (UniqueName: \"kubernetes.io/projected/403dedd4-49e9-43de-a391-d0da930a91fe-kube-api-access-lsnq8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:26:51.290332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.290300    6000 generic.go:356] "Generic (PLEG): container finished" podID="403dedd4-49e9-43de-a391-d0da930a91fe" containerID="fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad" exitCode=0
Nov 03 00:26:51.290501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.290344    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6" event={"ID":"403dedd4-49e9-43de-a391-d0da930a91fe","Type":"ContainerDied","Data":"fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad"}
Nov 03 00:26:51.290501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.290386    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6" event={"ID":"403dedd4-49e9-43de-a391-d0da930a91fe","Type":"ContainerDied","Data":"b8c46452c0795e98270ab1ad3f8730239a062a995a84977535b5471f6b1f4fed"}
Nov 03 00:26:51.290501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.290406    6000 scope.go:122] "RemoveContainer" containerID="fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad"
Nov 03 00:26:51.290603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.290538    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"
Nov 03 00:26:51.301624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.301599    6000 scope.go:122] "RemoveContainer" containerID="fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad"
Nov 03 00:26:51.301908 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:26:51.301883    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad\": container with ID starting with fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad not found: ID does not exist" containerID="fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad"
Nov 03 00:26:51.301995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.301917    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad"} err="failed to get container status \"fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad\": rpc error: code = NotFound desc = could not find container \"fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad\": container with ID starting with fd99bac71342bfa1c9c9e836bb646e88a87fdd2ed3d35454c3b0694899d661ad not found: ID does not exist"
Nov 03 00:26:51.341790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.341756    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"]
Nov 03 00:26:51.345272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:51.345231    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-f3806a-bcddc47c7-jfqh6"]
Nov 03 00:26:52.594810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:52.594770    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403dedd4-49e9-43de-a391-d0da930a91fe" path="/var/lib/kubelet/pods/403dedd4-49e9-43de-a391-d0da930a91fe/volumes"
Nov 03 00:26:55.301524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:55.300429    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:26:56.285352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:56.283304    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"]
Nov 03 00:26:56.285352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:26:56.283601    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy" containerID="cri-o://9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680" gracePeriod=30
Nov 03 00:27:01.515238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.515212    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:27:01.599721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.599633    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istiod-ca-cert\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.599905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.599724    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-credential-socket\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.599905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.599772    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-token\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-token\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.599905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.599813    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-envoy\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.600105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.599915    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-data\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.600105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.599962    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-socket\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.600105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.599988    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-podinfo\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.600293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.600137    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-certs\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.600293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.600268    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-kube-api-access-88n2h\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-kube-api-access-88n2h\") pod \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\" (UID: \"b8b6fc11-a279-4997-a8bf-c196b8650d8c\") "
Nov 03 00:27:01.600901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.600869    6000 generic.go:356] "Generic (PLEG): container finished" podID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerID="9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680" exitCode=0
Nov 03 00:27:01.601026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.601008    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" event={"ID":"b8b6fc11-a279-4997-a8bf-c196b8650d8c","Type":"ContainerDied","Data":"9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680"}
Nov 03 00:27:01.601092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.601062    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45" event={"ID":"b8b6fc11-a279-4997-a8bf-c196b8650d8c","Type":"ContainerDied","Data":"2493e2cc0b77e66b647b46610ceb755d8b8d94f3b80277bec205b7602a391cb5"}
Nov 03 00:27:01.601155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.601091    6000 scope.go:122] "RemoveContainer" containerID="9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680"
Nov 03 00:27:01.601545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.601352    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"
Nov 03 00:27:01.604130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.603816    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-kube-api-access-88n2h" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "kube-api-access-88n2h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:01.604237 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.604140    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-data" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:01.604605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.604456    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istiod-ca-cert" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:01.604605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.604523    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-socket" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:01.604605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.604584    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-certs" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:01.605163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.605135    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-credential-socket" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:01.606697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.606658    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-token" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:01.610425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.610387    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-envoy" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:01.612669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.612646    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-podinfo" pod "b8b6fc11-a279-4997-a8bf-c196b8650d8c" (UID: "b8b6fc11-a279-4997-a8bf-c196b8650d8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:01.654820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.654795    6000 scope.go:122] "RemoveContainer" containerID="daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135"
Nov 03 00:27:01.666045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.666020    6000 scope.go:122] "RemoveContainer" containerID="9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680"
Nov 03 00:27:01.666441 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:01.666415    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680\": container with ID starting with 9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680 not found: ID does not exist" containerID="9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680"
Nov 03 00:27:01.666562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.666448    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680"} err="failed to get container status \"9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680\": rpc error: code = NotFound desc = could not find container \"9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680\": container with ID starting with 9d5ced8bbd10e0f660238d2cf02b3dae661977ed9506c29ceadba01e41684680 not found: ID does not exist"
Nov 03 00:27:01.666562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.666469    6000 scope.go:122] "RemoveContainer" containerID="daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135"
Nov 03 00:27:01.666817 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:01.666774    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135\": container with ID starting with daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135 not found: ID does not exist" containerID="daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135"
Nov 03 00:27:01.666817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.666795    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135"} err="failed to get container status \"daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135\": rpc error: code = NotFound desc = could not find container \"daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135\": container with ID starting with daa0b59e67c0f80b982a7edb17267dbe18f957f94bbd9720df51b0dbc6c4b135 not found: ID does not exist"
Nov 03 00:27:01.703076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703034    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703069    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88n2h\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-kube-api-access-88n2h\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703088    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703105    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703120    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703134    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703147    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703159    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b8b6fc11-a279-4997-a8bf-c196b8650d8c-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.703288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.703169    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b8b6fc11-a279-4997-a8bf-c196b8650d8c-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:01.951073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.951030    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"]
Nov 03 00:27:01.954728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:01.954696    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f3806a-openshift-default-9d7696669-ltd45"]
Nov 03 00:27:02.576696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:02.576665    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" path="/var/lib/kubelet/pods/b8b6fc11-a279-4997-a8bf-c196b8650d8c/volumes"
Nov 03 00:27:02.616272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:02.616218    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"]
Nov 03 00:27:03.620219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:03.620180    6000 generic.go:356] "Generic (PLEG): container finished" podID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerID="86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3" exitCode=0
Nov 03 00:27:03.620922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:03.620301    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" event={"ID":"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb","Type":"ContainerDied","Data":"86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3"}
Nov 03 00:27:03.620922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:03.620786    6000 scope.go:122] "RemoveContainer" containerID="86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3"
Nov 03 00:27:03.622660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:03.622625    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:27:03.622783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:03.622696    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:27:03.622783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:03.622737    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:27:04.632285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:04.632228    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" event={"ID":"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb","Type":"ContainerStarted","Data":"546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5"}
Nov 03 00:27:04.744942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:04.744911    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:27:04.746363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:04.746339    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:27:04.746476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:04.746389    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 00:27:05.744916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:05.744881    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:27:05.745337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:05.744942    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 00:27:06.269443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:06.269402    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:27:06.745271 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:06.745218    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:27:06.745679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:06.745321    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 00:27:07.617567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.617527    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w" podUID="f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" containerName="payload-processing" containerID="cri-o://dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3" gracePeriod=25
Nov 03 00:27:07.745587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.745551    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:27:07.746001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.745644    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.925219    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"]
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.926549    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy"
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.926568    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy"
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.926646    6000 cpu_manager.go:414] "containerMap: removing container" podUID="403dedd4-49e9-43de-a391-d0da930a91fe" containerName="maas-api"
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.926655    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403dedd4-49e9-43de-a391-d0da930a91fe" containerName="maas-api"
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.927013    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy"
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.927033    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy"
Nov 03 00:27:07.927757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.927060    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="403dedd4-49e9-43de-a391-d0da930a91fe" containerName="maas-api"
Nov 03 00:27:07.937690 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.936959    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"]
Nov 03 00:27:07.937690 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.937073    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:07.953224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.953189    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-dda6df-serving-cert\""
Nov 03 00:27:07.965715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.965177    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-metrics-certs\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:07.965715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.965280    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:07.965715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.965434    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:07.965715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.965477    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgbxm\" (UniqueName: \"kubernetes.io/projected/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-kube-api-access-cgbxm\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:07.970068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:07.970044    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:27:08.066951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.066849    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-kube-api-access-dbbvv\" (UniqueName: \"kubernetes.io/projected/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-kube-api-access-dbbvv\") pod \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") "
Nov 03 00:27:08.067223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.067200    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-plugins-config-volume\") pod \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\" (UID: \"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42\") "
Nov 03 00:27:08.067646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.067622    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-plugins-config-volume" pod "f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" (UID: "f516bf3a-d7a5-4653-80f3-f17fb2ed9b42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:08.067769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.067755    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-metrics-certs\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.067809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.067788    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.067925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.067910    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.067977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.067942    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgbxm\" (UniqueName: \"kubernetes.io/projected/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-kube-api-access-cgbxm\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.068312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.068292    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:08.068810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.068789    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.069357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.069334    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-kube-api-access-dbbvv" pod "f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" (UID: "f516bf3a-d7a5-4653-80f3-f17fb2ed9b42"). InnerVolumeSpecName "kube-api-access-dbbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:08.070411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.070394    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-maas-api-tls\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.070510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.070451    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-metrics-certs\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.096357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.096246    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgbxm\" (UniqueName: \"kubernetes.io/projected/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-kube-api-access-cgbxm\") pod \"maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.183336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.175075    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbbvv\" (UniqueName: \"kubernetes.io/projected/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42-kube-api-access-dbbvv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:08.290283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.284506    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:08.667620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.666420    6000 generic.go:356] "Generic (PLEG): container finished" podID="f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" containerID="dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3" exitCode=0
Nov 03 00:27:08.667620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.666474    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"
Nov 03 00:27:08.667620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.666505    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w" event={"ID":"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42","Type":"ContainerDied","Data":"dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3"}
Nov 03 00:27:08.667620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.666567    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w" event={"ID":"f516bf3a-d7a5-4653-80f3-f17fb2ed9b42","Type":"ContainerDied","Data":"f855381de3315cb7393ecf01907887ecdf92b2922bf521954481dac50482c2d5"}
Nov 03 00:27:08.667620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.666602    6000 scope.go:122] "RemoveContainer" containerID="dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3"
Nov 03 00:27:08.667620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.667310    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"]
Nov 03 00:27:08.674985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.674967    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:27:08.722206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.722183    6000 scope.go:122] "RemoveContainer" containerID="dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3"
Nov 03 00:27:08.722723 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:08.722697    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3\": container with ID starting with dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3 not found: ID does not exist" containerID="dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3"
Nov 03 00:27:08.722811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.722729    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3"} err="failed to get container status \"dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3\": rpc error: code = NotFound desc = could not find container \"dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3\": container with ID starting with dcc5144917abbf9b5a9c66a7d2d54d9785538d12b5ebe789ffbb7191ef86ecb3 not found: ID does not exist"
Nov 03 00:27:08.741021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.740992    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"]
Nov 03 00:27:08.745121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.745091    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:27:08.745280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.745147    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 00:27:08.745398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:08.745379    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-cb05ae-5d86698897-qhl2w"]
Nov 03 00:27:09.676078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:09.675979    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5" event={"ID":"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427","Type":"ContainerStarted","Data":"fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235"}
Nov 03 00:27:09.676078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:09.676035    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5" event={"ID":"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427","Type":"ContainerStarted","Data":"7bc65c1b9a16ff4aaf9ccf07a43de5b1951dbe7ebc6e8cbc3962719058e5fe3e"}
Nov 03 00:27:09.676618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:09.676191    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:09.716707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:09.716653    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5" podStartSLOduration=2.2725608250000002 podStartE2EDuration="2.71663819s" totalImagesPullingTime="444.077365ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:27:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:27:09.713138472 +0000 UTC m=+1267.268490498" watchObservedRunningTime="2026-11-03 00:27:09.71663819 +0000 UTC m=+1267.271990215"
Nov 03 00:27:09.745574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:09.745528    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused" start-of-body=
Nov 03 00:27:09.745787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:09.745591    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.108:15021/healthz/ready\": dial tcp 10.134.0.108:15021: connect: connection refused"
Nov 03 00:27:10.569109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:10.569070    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" path="/var/lib/kubelet/pods/f516bf3a-d7a5-4653-80f3-f17fb2ed9b42/volumes"
Nov 03 00:27:10.749365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:10.749339    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:27:10.750388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:10.750367    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:27:15.054343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:15.054110    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:27:15.740464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:15.740433    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:27:19.935076 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:19.935041    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:27:25.440698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:25.440654    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"]
Nov 03 00:27:25.479676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:25.479630    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"]
Nov 03 00:27:27.744779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:27.744745    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"]
Nov 03 00:27:27.745321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:27.744928    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz" podUID="f4609dd8-273d-43a9-94ac-70e51d339fa2" containerName="maas-api" containerID="cri-o://2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47" gracePeriod=30
Nov 03 00:27:28.202409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.196888    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"]
Nov 03 00:27:28.202409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.197121    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf" podUID="7014038a-0215-4164-b032-feff0b68056f" containerName="maas-api" containerID="cri-o://e5e822a958f149fab80a7b675a95a4d06ffdfb833d9cfd3448bbc0aa10d165db" gracePeriod=30
Nov 03 00:27:28.443232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.443150    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:27:28.621299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.621202    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-maas-api-tls\") pod \"f4609dd8-273d-43a9-94ac-70e51d339fa2\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") "
Nov 03 00:27:28.631939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.625329    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-maas-api-tls" pod "f4609dd8-273d-43a9-94ac-70e51d339fa2" (UID: "f4609dd8-273d-43a9-94ac-70e51d339fa2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:27:28.741279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.725442    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-metrics-certs\") pod \"f4609dd8-273d-43a9-94ac-70e51d339fa2\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") "
Nov 03 00:27:28.741279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.725507    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4609dd8-273d-43a9-94ac-70e51d339fa2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4609dd8-273d-43a9-94ac-70e51d339fa2-openshift-service-ca\") pod \"f4609dd8-273d-43a9-94ac-70e51d339fa2\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") "
Nov 03 00:27:28.741279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.725544    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4609dd8-273d-43a9-94ac-70e51d339fa2-kube-api-access-v7v7k\" (UniqueName: \"kubernetes.io/projected/f4609dd8-273d-43a9-94ac-70e51d339fa2-kube-api-access-v7v7k\") pod \"f4609dd8-273d-43a9-94ac-70e51d339fa2\" (UID: \"f4609dd8-273d-43a9-94ac-70e51d339fa2\") "
Nov 03 00:27:28.741279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.726329    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:28.741279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.728014    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4609dd8-273d-43a9-94ac-70e51d339fa2-openshift-service-ca" pod "f4609dd8-273d-43a9-94ac-70e51d339fa2" (UID: "f4609dd8-273d-43a9-94ac-70e51d339fa2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:28.768934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.767105    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4609dd8-273d-43a9-94ac-70e51d339fa2-kube-api-access-v7v7k" pod "f4609dd8-273d-43a9-94ac-70e51d339fa2" (UID: "f4609dd8-273d-43a9-94ac-70e51d339fa2"). InnerVolumeSpecName "kube-api-access-v7v7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:28.850281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.811831    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-metrics-certs" pod "f4609dd8-273d-43a9-94ac-70e51d339fa2" (UID: "f4609dd8-273d-43a9-94ac-70e51d339fa2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:27:28.850281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.828677    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f4609dd8-273d-43a9-94ac-70e51d339fa2-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:28.850281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.828706    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4609dd8-273d-43a9-94ac-70e51d339fa2-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:28.850281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.828725    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7v7k\" (UniqueName: \"kubernetes.io/projected/f4609dd8-273d-43a9-94ac-70e51d339fa2-kube-api-access-v7v7k\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:28.909299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.908072    6000 generic.go:356] "Generic (PLEG): container finished" podID="7014038a-0215-4164-b032-feff0b68056f" containerID="e5e822a958f149fab80a7b675a95a4d06ffdfb833d9cfd3448bbc0aa10d165db" exitCode=0
Nov 03 00:27:28.909299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.908294    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf" event={"ID":"7014038a-0215-4164-b032-feff0b68056f","Type":"ContainerDied","Data":"e5e822a958f149fab80a7b675a95a4d06ffdfb833d9cfd3448bbc0aa10d165db"}
Nov 03 00:27:28.924427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.922558    6000 generic.go:356] "Generic (PLEG): container finished" podID="f4609dd8-273d-43a9-94ac-70e51d339fa2" containerID="2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47" exitCode=0
Nov 03 00:27:28.924427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.922644    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz" event={"ID":"f4609dd8-273d-43a9-94ac-70e51d339fa2","Type":"ContainerDied","Data":"2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47"}
Nov 03 00:27:28.924427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.922695    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz" event={"ID":"f4609dd8-273d-43a9-94ac-70e51d339fa2","Type":"ContainerDied","Data":"f280440f59d0a81332106d4a9e1c89beaeb13203237e6859c19ce8e6738916b0"}
Nov 03 00:27:28.924427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.922725    6000 scope.go:122] "RemoveContainer" containerID="2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47"
Nov 03 00:27:28.924427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:28.922920    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"
Nov 03 00:27:29.026880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.026855    6000 scope.go:122] "RemoveContainer" containerID="2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47"
Nov 03 00:27:29.027324 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:29.027303    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47\": container with ID starting with 2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47 not found: ID does not exist" containerID="2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47"
Nov 03 00:27:29.027448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.027431    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47"} err="failed to get container status \"2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47\": rpc error: code = NotFound desc = could not find container \"2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47\": container with ID starting with 2964f64522b34e493499c7485eacb7798cd479199229e17a1cd5ef01fe2e1e47 not found: ID does not exist"
Nov 03 00:27:29.064811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.064776    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"]
Nov 03 00:27:29.071778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.071751    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-cb05ae-75b8f7c9dd-mscqz"]
Nov 03 00:27:29.133792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.133764    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:27:29.148349 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.148322    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-metrics-certs\") pod \"7014038a-0215-4164-b032-feff0b68056f\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") "
Nov 03 00:27:29.148459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.148355    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7014038a-0215-4164-b032-feff0b68056f-kube-api-access-6gbnq\" (UniqueName: \"kubernetes.io/projected/7014038a-0215-4164-b032-feff0b68056f-kube-api-access-6gbnq\") pod \"7014038a-0215-4164-b032-feff0b68056f\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") "
Nov 03 00:27:29.148513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.148506    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7014038a-0215-4164-b032-feff0b68056f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7014038a-0215-4164-b032-feff0b68056f-openshift-service-ca\") pod \"7014038a-0215-4164-b032-feff0b68056f\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") "
Nov 03 00:27:29.148640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.148627    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls\") pod \"7014038a-0215-4164-b032-feff0b68056f\" (UID: \"7014038a-0215-4164-b032-feff0b68056f\") "
Nov 03 00:27:29.148856 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.148829    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7014038a-0215-4164-b032-feff0b68056f-openshift-service-ca" pod "7014038a-0215-4164-b032-feff0b68056f" (UID: "7014038a-0215-4164-b032-feff0b68056f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:29.149415 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.149395    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7014038a-0215-4164-b032-feff0b68056f-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:29.150749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.150721    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7014038a-0215-4164-b032-feff0b68056f-kube-api-access-6gbnq" pod "7014038a-0215-4164-b032-feff0b68056f" (UID: "7014038a-0215-4164-b032-feff0b68056f"). InnerVolumeSpecName "kube-api-access-6gbnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:29.150749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.150721    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls" pod "7014038a-0215-4164-b032-feff0b68056f" (UID: "7014038a-0215-4164-b032-feff0b68056f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:27:29.150903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.150781    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-metrics-certs" pod "7014038a-0215-4164-b032-feff0b68056f" (UID: "7014038a-0215-4164-b032-feff0b68056f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:27:29.250957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.250877    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:29.250957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.250912    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7014038a-0215-4164-b032-feff0b68056f-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:29.250957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.250923    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gbnq\" (UniqueName: \"kubernetes.io/projected/7014038a-0215-4164-b032-feff0b68056f-kube-api-access-6gbnq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:29.930639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.930605    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"
Nov 03 00:27:29.931141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.930669    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf" event={"ID":"7014038a-0215-4164-b032-feff0b68056f","Type":"ContainerDied","Data":"35a8d181b53c07d5a72923664065a224219d022644d822ab9352e590085dd6a5"}
Nov 03 00:27:29.931141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.930739    6000 scope.go:122] "RemoveContainer" containerID="e5e822a958f149fab80a7b675a95a4d06ffdfb833d9cfd3448bbc0aa10d165db"
Nov 03 00:27:29.982021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.981988    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"]
Nov 03 00:27:29.993712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:29.993685    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-493ec8-587fdf75cf-wgqdf"]
Nov 03 00:27:30.441578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.441520    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z" podUID="a6765be6-efe8-457f-be3d-3e2fd547684d" containerName="payload-processing" containerID="cri-o://ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59" gracePeriod=25
Nov 03 00:27:30.480642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.480605    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r" podUID="0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" containerName="payload-pre-processing" containerID="cri-o://71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d" gracePeriod=25
Nov 03 00:27:30.568079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.568047    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7014038a-0215-4164-b032-feff0b68056f" path="/var/lib/kubelet/pods/7014038a-0215-4164-b032-feff0b68056f/volumes"
Nov 03 00:27:30.568575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.568551    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4609dd8-273d-43a9-94ac-70e51d339fa2" path="/var/lib/kubelet/pods/f4609dd8-273d-43a9-94ac-70e51d339fa2/volumes"
Nov 03 00:27:30.787772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.787745    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:27:30.824661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.824637    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:27:30.945134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.945034    6000 generic.go:356] "Generic (PLEG): container finished" podID="0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" containerID="71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d" exitCode=0
Nov 03 00:27:30.945134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.945098    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"
Nov 03 00:27:30.945134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.945115    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r" event={"ID":"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf","Type":"ContainerDied","Data":"71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d"}
Nov 03 00:27:30.945859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.945166    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r" event={"ID":"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf","Type":"ContainerDied","Data":"c6e2d8df581c381f44eea424008176c9437147542082e00cf8ce6a2f34c625b2"}
Nov 03 00:27:30.945859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.945191    6000 scope.go:122] "RemoveContainer" containerID="71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d"
Nov 03 00:27:30.947202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.947178    6000 generic.go:356] "Generic (PLEG): container finished" podID="a6765be6-efe8-457f-be3d-3e2fd547684d" containerID="ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59" exitCode=0
Nov 03 00:27:30.947347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.947272    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z" event={"ID":"a6765be6-efe8-457f-be3d-3e2fd547684d","Type":"ContainerDied","Data":"ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59"}
Nov 03 00:27:30.947347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.947325    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z" event={"ID":"a6765be6-efe8-457f-be3d-3e2fd547684d","Type":"ContainerDied","Data":"168e4b9a3526c034ce0d0348afa2887ff20676242b15f6c28b10af8646f4d544"}
Nov 03 00:27:30.947498 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.947482    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"
Nov 03 00:27:30.955658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.955638    6000 scope.go:122] "RemoveContainer" containerID="71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d"
Nov 03 00:27:30.955911 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:30.955895    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d\": container with ID starting with 71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d not found: ID does not exist" containerID="71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d"
Nov 03 00:27:30.955951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.955922    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d"} err="failed to get container status \"71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d\": rpc error: code = NotFound desc = could not find container \"71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d\": container with ID starting with 71bfe5590e19590e5f64a359ccdb87717bd6e29ed40ec07c8bb0a4ea969d687d not found: ID does not exist"
Nov 03 00:27:30.955951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.955939    6000 scope.go:122] "RemoveContainer" containerID="ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59"
Nov 03 00:27:30.965826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.965802    6000 scope.go:122] "RemoveContainer" containerID="ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59"
Nov 03 00:27:30.973659 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:30.966867    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59\": container with ID starting with ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59 not found: ID does not exist" containerID="ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59"
Nov 03 00:27:30.973659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.966939    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59"} err="failed to get container status \"ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59\": rpc error: code = NotFound desc = could not find container \"ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59\": container with ID starting with ac7df25065d03614fa83ed28c630123b7edd50c027bd99e03679b1b9fcb61e59 not found: ID does not exist"
Nov 03 00:27:30.985045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.981574    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-kube-api-access-l5ks8\" (UniqueName: \"kubernetes.io/projected/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-kube-api-access-l5ks8\") pod \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") "
Nov 03 00:27:30.985045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.981738    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6765be6-efe8-457f-be3d-3e2fd547684d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6765be6-efe8-457f-be3d-3e2fd547684d-plugins-config-volume\") pod \"a6765be6-efe8-457f-be3d-3e2fd547684d\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") "
Nov 03 00:27:30.985045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.981792    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6765be6-efe8-457f-be3d-3e2fd547684d-kube-api-access-d6chh\" (UniqueName: \"kubernetes.io/projected/a6765be6-efe8-457f-be3d-3e2fd547684d-kube-api-access-d6chh\") pod \"a6765be6-efe8-457f-be3d-3e2fd547684d\" (UID: \"a6765be6-efe8-457f-be3d-3e2fd547684d\") "
Nov 03 00:27:30.985045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.982029    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-plugins-config-volume\") pod \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\" (UID: \"0aba12fe-31b5-4eba-b3be-fdccd6b3cedf\") "
Nov 03 00:27:30.985413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.985280    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-plugins-config-volume" pod "0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" (UID: "0aba12fe-31b5-4eba-b3be-fdccd6b3cedf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:31.004273 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:30.986121    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6765be6-efe8-457f-be3d-3e2fd547684d-plugins-config-volume" pod "a6765be6-efe8-457f-be3d-3e2fd547684d" (UID: "a6765be6-efe8-457f-be3d-3e2fd547684d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:31.041353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.038569    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-kube-api-access-l5ks8" pod "0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" (UID: "0aba12fe-31b5-4eba-b3be-fdccd6b3cedf"). InnerVolumeSpecName "kube-api-access-l5ks8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:31.046499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.046455    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6765be6-efe8-457f-be3d-3e2fd547684d-kube-api-access-d6chh" pod "a6765be6-efe8-457f-be3d-3e2fd547684d" (UID: "a6765be6-efe8-457f-be3d-3e2fd547684d"). InnerVolumeSpecName "kube-api-access-d6chh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:31.088513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.087165    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a6765be6-efe8-457f-be3d-3e2fd547684d-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:31.088513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.087196    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6chh\" (UniqueName: \"kubernetes.io/projected/a6765be6-efe8-457f-be3d-3e2fd547684d-kube-api-access-d6chh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:31.088513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.087205    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:31.088513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.087214    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5ks8\" (UniqueName: \"kubernetes.io/projected/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf-kube-api-access-l5ks8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:31.289271 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.289233    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"]
Nov 03 00:27:31.296123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.296096    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-493ec8-5c8db9755f-b488r"]
Nov 03 00:27:31.327842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.327810    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"]
Nov 03 00:27:31.330980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:31.330956    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-493ec8-67976d6b7f-9p85z"]
Nov 03 00:27:32.568671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:32.568639    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" path="/var/lib/kubelet/pods/0aba12fe-31b5-4eba-b3be-fdccd6b3cedf/volumes"
Nov 03 00:27:32.569097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:32.569083    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6765be6-efe8-457f-be3d-3e2fd547684d" path="/var/lib/kubelet/pods/a6765be6-efe8-457f-be3d-3e2fd547684d/volumes"
Nov 03 00:27:39.575337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:39.575283    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"]
Nov 03 00:27:44.576809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:44.576772    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8" podUID="aa0de994-cdee-4f27-b8ca-860b8e31b169" containerName="payload-pre-processing" containerID="cri-o://add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4" gracePeriod=25
Nov 03 00:27:44.864374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:44.860268    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:27:44.955635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:44.955602    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa0de994-cdee-4f27-b8ca-860b8e31b169-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0de994-cdee-4f27-b8ca-860b8e31b169-plugins-config-volume\") pod \"aa0de994-cdee-4f27-b8ca-860b8e31b169\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") "
Nov 03 00:27:44.955892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:44.955873    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa0de994-cdee-4f27-b8ca-860b8e31b169-kube-api-access-s6cbd\" (UniqueName: \"kubernetes.io/projected/aa0de994-cdee-4f27-b8ca-860b8e31b169-kube-api-access-s6cbd\") pod \"aa0de994-cdee-4f27-b8ca-860b8e31b169\" (UID: \"aa0de994-cdee-4f27-b8ca-860b8e31b169\") "
Nov 03 00:27:44.956427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:44.956129    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa0de994-cdee-4f27-b8ca-860b8e31b169-plugins-config-volume" pod "aa0de994-cdee-4f27-b8ca-860b8e31b169" (UID: "aa0de994-cdee-4f27-b8ca-860b8e31b169"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:44.956634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:44.956616    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa0de994-cdee-4f27-b8ca-860b8e31b169-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:44.958621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:44.958587    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa0de994-cdee-4f27-b8ca-860b8e31b169-kube-api-access-s6cbd" pod "aa0de994-cdee-4f27-b8ca-860b8e31b169" (UID: "aa0de994-cdee-4f27-b8ca-860b8e31b169"). InnerVolumeSpecName "kube-api-access-s6cbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:45.068280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.061667    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6cbd\" (UniqueName: \"kubernetes.io/projected/aa0de994-cdee-4f27-b8ca-860b8e31b169-kube-api-access-s6cbd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:45.266307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.264354    6000 generic.go:356] "Generic (PLEG): container finished" podID="aa0de994-cdee-4f27-b8ca-860b8e31b169" containerID="add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4" exitCode=0
Nov 03 00:27:45.266307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.264626    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8" event={"ID":"aa0de994-cdee-4f27-b8ca-860b8e31b169","Type":"ContainerDied","Data":"add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4"}
Nov 03 00:27:45.266307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.264677    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8" event={"ID":"aa0de994-cdee-4f27-b8ca-860b8e31b169","Type":"ContainerDied","Data":"279e1aad48dfa130cd580571db55ddaeb928926590f2756c2ef114add986c140"}
Nov 03 00:27:45.266307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.264705    6000 scope.go:122] "RemoveContainer" containerID="add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4"
Nov 03 00:27:45.266307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.264855    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"
Nov 03 00:27:45.277772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.277744    6000 scope.go:122] "RemoveContainer" containerID="add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4"
Nov 03 00:27:45.278060 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:45.278036    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4\": container with ID starting with add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4 not found: ID does not exist" containerID="add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4"
Nov 03 00:27:45.278135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.278070    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4"} err="failed to get container status \"add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4\": rpc error: code = NotFound desc = could not find container \"add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4\": container with ID starting with add3d7c2af5a5e9e6d1a64b6006cc9b445b11a1a1dd88d2086b897972cd149e4 not found: ID does not exist"
Nov 03 00:27:45.317512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.317478    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"]
Nov 03 00:27:45.329170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:45.329143    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d43409-66cd8996c5-kc6t8"]
Nov 03 00:27:46.569755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:46.569227    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa0de994-cdee-4f27-b8ca-860b8e31b169" path="/var/lib/kubelet/pods/aa0de994-cdee-4f27-b8ca-860b8e31b169/volumes"
Nov 03 00:27:49.619974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:49.619938    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"]
Nov 03 00:27:49.620443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:49.620167    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy" containerID="cri-o://65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b" gracePeriod=30
Nov 03 00:27:49.632406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:49.632333    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"]
Nov 03 00:27:49.632684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:49.632658    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy" containerID="cri-o://bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764" gracePeriod=30
Nov 03 00:27:49.636926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:49.636898    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"]
Nov 03 00:27:49.637150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:49.637107    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy" containerID="cri-o://a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605" gracePeriod=30
Nov 03 00:27:49.897991 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:49.897903    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:27:54.977466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:54.977435    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:27:54.982244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:54.982215    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:27:55.005714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.005690    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:27:55.134762 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.134722    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-envoy\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.134966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.134802    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-envoy\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.134966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.134846    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35ceba1e-6211-4524-9062-037a85917c22-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ceba1e-6211-4524-9062-037a85917c22-istio-podinfo\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.134966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.134887    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-socket\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.135141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.134969    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-token\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-token\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.135141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135014    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-istio-token\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-istio-token\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.135141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135089    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istiod-ca-cert\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.135141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135134    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-podinfo\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.135408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135167    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-credential-socket\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.135408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135320    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-envoy\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.135408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135350    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-data\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.135408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135390    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-credential-socket\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.135600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135423    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-data\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.135600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135488    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35ceba1e-6211-4524-9062-037a85917c22-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ceba1e-6211-4524-9062-037a85917c22-istiod-ca-cert\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.135600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135572    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-kube-api-access-rkr9f\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-kube-api-access-rkr9f\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.135757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135601    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-token\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-token\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.135757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135668    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-certs\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.135757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135720    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-kube-api-access-brhjx\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-kube-api-access-brhjx\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.135901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.135754    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-socket\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.136300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.136087    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-credential-socket" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.136552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.136485    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-socket" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.136552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.136518    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istiod-ca-cert" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:55.137234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137001    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-kube-api-access-pmzzt\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-kube-api-access-pmzzt\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.137234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137063    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-socket\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.137234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137073    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35ceba1e-6211-4524-9062-037a85917c22-istiod-ca-cert" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:55.137234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137101    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-credential-socket" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.137234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137111    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-certs\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.137234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137172    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-certs\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.137611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137318    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-certs" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.137611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137329    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-certs" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.137611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137376    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-data" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.137611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137524    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-socket" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.138896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137869    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istiod-ca-cert\") pod \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\" (UID: \"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50\") "
Nov 03 00:27:55.138896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137934    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-data\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.138896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.137993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-credential-socket\") pod \"35ceba1e-6211-4524-9062-037a85917c22\" (UID: \"35ceba1e-6211-4524-9062-037a85917c22\") "
Nov 03 00:27:55.138896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.138014    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-data" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.138896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.138027    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-podinfo\") pod \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\" (UID: \"529a05c4-4cd7-4729-bcb9-70cfa9b6d448\") "
Nov 03 00:27:55.138896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.138713    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istiod-ca-cert" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:27:55.138896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.138718    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-data" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.138997    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-certs" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139011    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139031    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139046    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139059    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139072    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139087    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139100    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139114    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139129    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139143    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139156    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35ceba1e-6211-4524-9062-037a85917c22-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139168    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.139308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139236    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-socket" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.140006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139389    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-credential-socket" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.140006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.139837    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-token" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:55.140108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.140028    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-envoy" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.143831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.143511    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-kube-api-access-pmzzt" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "kube-api-access-pmzzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:55.143831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.143605    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-kube-api-access-rkr9f" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "kube-api-access-rkr9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:55.143831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.143695    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-kube-api-access-brhjx" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "kube-api-access-brhjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:55.143831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.143780    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35ceba1e-6211-4524-9062-037a85917c22-istio-podinfo" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:55.143831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.143797    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-podinfo" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:55.144164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.143897    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-envoy" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.144164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.144148    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-token" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:55.144402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.144377    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-istio-token" pod "35ceba1e-6211-4524-9062-037a85917c22" (UID: "35ceba1e-6211-4524-9062-037a85917c22"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:27:55.144936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.144919    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-podinfo" pod "529a05c4-4cd7-4729-bcb9-70cfa9b6d448" (UID: "529a05c4-4cd7-4729-bcb9-70cfa9b6d448"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:27:55.145422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.145403    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-envoy" pod "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" (UID: "9d22b28f-afb0-4b8c-96a8-31e7cdb63c50"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240339    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240365    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240374    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240384    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240392    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkr9f\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-kube-api-access-rkr9f\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240403    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240411    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brhjx\" (UniqueName: \"kubernetes.io/projected/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-kube-api-access-brhjx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240420    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmzzt\" (UniqueName: \"kubernetes.io/projected/35ceba1e-6211-4524-9062-037a85917c22-kube-api-access-pmzzt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.240425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240428    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.241018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240436    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.241018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240444    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.241018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240453    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.241018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240462    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35ceba1e-6211-4524-9062-037a85917c22-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.241018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240469    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/529a05c4-4cd7-4729-bcb9-70cfa9b6d448-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.241018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.240478    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35ceba1e-6211-4524-9062-037a85917c22-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:27:55.516468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.516387    6000 generic.go:356] "Generic (PLEG): container finished" podID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerID="a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605" exitCode=0
Nov 03 00:27:55.516468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.516451    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"
Nov 03 00:27:55.516682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.516452    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" event={"ID":"529a05c4-4cd7-4729-bcb9-70cfa9b6d448","Type":"ContainerDied","Data":"a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605"}
Nov 03 00:27:55.516682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.516596    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9" event={"ID":"529a05c4-4cd7-4729-bcb9-70cfa9b6d448","Type":"ContainerDied","Data":"bae13a46614dc3f26c21ffb4a393074f891beed3470941dcd3fb0ec986f61b33"}
Nov 03 00:27:55.516682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.516627    6000 scope.go:122] "RemoveContainer" containerID="a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605"
Nov 03 00:27:55.518035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.518009    6000 generic.go:356] "Generic (PLEG): container finished" podID="35ceba1e-6211-4524-9062-037a85917c22" containerID="bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764" exitCode=0
Nov 03 00:27:55.518149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.518058    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"
Nov 03 00:27:55.518149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.518098    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" event={"ID":"35ceba1e-6211-4524-9062-037a85917c22","Type":"ContainerDied","Data":"bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764"}
Nov 03 00:27:55.518149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.518142    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq" event={"ID":"35ceba1e-6211-4524-9062-037a85917c22","Type":"ContainerDied","Data":"e0126dde8bf17fd1aaf940319f5f6b89ffc18c1f3e6358e03187dba1984ec7e4"}
Nov 03 00:27:55.519730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.519691    6000 generic.go:356] "Generic (PLEG): container finished" podID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerID="65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b" exitCode=0
Nov 03 00:27:55.519730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.519716    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" event={"ID":"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50","Type":"ContainerDied","Data":"65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b"}
Nov 03 00:27:55.519919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.519739    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"
Nov 03 00:27:55.519919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.519765    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk" event={"ID":"9d22b28f-afb0-4b8c-96a8-31e7cdb63c50","Type":"ContainerDied","Data":"1b0febe950669a8226920dc3bf1dc803bf535bfa358b9190e6f27ae227965201"}
Nov 03 00:27:55.527752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.527734    6000 scope.go:122] "RemoveContainer" containerID="a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605"
Nov 03 00:27:55.528181 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:55.528155    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605\": container with ID starting with a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605 not found: ID does not exist" containerID="a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605"
Nov 03 00:27:55.528272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.528194    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605"} err="failed to get container status \"a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605\": rpc error: code = NotFound desc = could not find container \"a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605\": container with ID starting with a8efb5ff5ccc22c856a8372e177fc91347ad973f92f484bf70fbc795fe0ad605 not found: ID does not exist"
Nov 03 00:27:55.528272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.528219    6000 scope.go:122] "RemoveContainer" containerID="bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764"
Nov 03 00:27:55.538696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.538681    6000 scope.go:122] "RemoveContainer" containerID="bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764"
Nov 03 00:27:55.538929 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:55.538904    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764\": container with ID starting with bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764 not found: ID does not exist" containerID="bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764"
Nov 03 00:27:55.538996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.538936    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764"} err="failed to get container status \"bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764\": rpc error: code = NotFound desc = could not find container \"bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764\": container with ID starting with bb28a74bbc24692812c530d75c33129d154711f72bc5bacb6d38b9e9b6592764 not found: ID does not exist"
Nov 03 00:27:55.538996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.538953    6000 scope.go:122] "RemoveContainer" containerID="65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b"
Nov 03 00:27:55.548561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.548545    6000 scope.go:122] "RemoveContainer" containerID="65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b"
Nov 03 00:27:55.548813 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:27:55.548795    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b\": container with ID starting with 65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b not found: ID does not exist" containerID="65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b"
Nov 03 00:27:55.548878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.548819    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b"} err="failed to get container status \"65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b\": rpc error: code = NotFound desc = could not find container \"65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b\": container with ID starting with 65e428e5193a81c5e2efd474df5781246a8b8b1b1bc5c19aa96ced5125455f1b not found: ID does not exist"
Nov 03 00:27:55.578854 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.578831    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"]
Nov 03 00:27:55.590106 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.590086    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-qfzr9"]
Nov 03 00:27:55.621839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.621811    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"]
Nov 03 00:27:55.632011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.631982    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-q6nlk"]
Nov 03 00:27:55.663438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.663417    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"]
Nov 03 00:27:55.669650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:55.669630    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-pbsmq"]
Nov 03 00:27:56.567356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:56.567323    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35ceba1e-6211-4524-9062-037a85917c22" path="/var/lib/kubelet/pods/35ceba1e-6211-4524-9062-037a85917c22/volumes"
Nov 03 00:27:56.567792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:56.567778    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" path="/var/lib/kubelet/pods/529a05c4-4cd7-4729-bcb9-70cfa9b6d448/volumes"
Nov 03 00:27:56.568321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:27:56.568297    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" path="/var/lib/kubelet/pods/9d22b28f-afb0-4b8c-96a8-31e7cdb63c50/volumes"
Nov 03 00:28:00.251794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.251761    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"]
Nov 03 00:28:00.252239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.251977    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v" podUID="a44e2389-13b8-4509-89e0-409888b5edb8" containerName="maas-api" containerID="cri-o://a1346a3c27899c9a7f87d6856047071ed931108832eb7b84ffd42f0dec01603d" gracePeriod=30
Nov 03 00:28:00.826341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.823598    6000 generic.go:356] "Generic (PLEG): container finished" podID="a44e2389-13b8-4509-89e0-409888b5edb8" containerID="a1346a3c27899c9a7f87d6856047071ed931108832eb7b84ffd42f0dec01603d" exitCode=0
Nov 03 00:28:00.826341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.823789    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v" event={"ID":"a44e2389-13b8-4509-89e0-409888b5edb8","Type":"ContainerDied","Data":"a1346a3c27899c9a7f87d6856047071ed931108832eb7b84ffd42f0dec01603d"}
Nov 03 00:28:00.826341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.823851    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v" event={"ID":"a44e2389-13b8-4509-89e0-409888b5edb8","Type":"ContainerDied","Data":"0d892919f2f67f77e26927cd60c5797c6297ee0f68bcccd447ce02e93e8a91d6"}
Nov 03 00:28:00.826341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.823874    6000 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d892919f2f67f77e26927cd60c5797c6297ee0f68bcccd447ce02e93e8a91d6"
Nov 03 00:28:00.826341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.824406    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:28:00.932990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.932923    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44e2389-13b8-4509-89e0-409888b5edb8-kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/a44e2389-13b8-4509-89e0-409888b5edb8-kube-api-access-qnr5n\") pod \"a44e2389-13b8-4509-89e0-409888b5edb8\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") "
Nov 03 00:28:00.933555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.933526    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-maas-api-tls\") pod \"a44e2389-13b8-4509-89e0-409888b5edb8\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") "
Nov 03 00:28:00.933727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.933698    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44e2389-13b8-4509-89e0-409888b5edb8-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44e2389-13b8-4509-89e0-409888b5edb8-openshift-service-ca\") pod \"a44e2389-13b8-4509-89e0-409888b5edb8\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") "
Nov 03 00:28:00.933822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.933760    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-metrics-certs\") pod \"a44e2389-13b8-4509-89e0-409888b5edb8\" (UID: \"a44e2389-13b8-4509-89e0-409888b5edb8\") "
Nov 03 00:28:00.939442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.939414    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-metrics-certs" pod "a44e2389-13b8-4509-89e0-409888b5edb8" (UID: "a44e2389-13b8-4509-89e0-409888b5edb8"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:00.939561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.939452    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44e2389-13b8-4509-89e0-409888b5edb8-openshift-service-ca" pod "a44e2389-13b8-4509-89e0-409888b5edb8" (UID: "a44e2389-13b8-4509-89e0-409888b5edb8"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:00.940880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.940851    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44e2389-13b8-4509-89e0-409888b5edb8-kube-api-access-qnr5n" pod "a44e2389-13b8-4509-89e0-409888b5edb8" (UID: "a44e2389-13b8-4509-89e0-409888b5edb8"). InnerVolumeSpecName "kube-api-access-qnr5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:00.942159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:00.942130    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-maas-api-tls" pod "a44e2389-13b8-4509-89e0-409888b5edb8" (UID: "a44e2389-13b8-4509-89e0-409888b5edb8"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:01.035981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:01.035949    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:01.035981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:01.035978    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnr5n\" (UniqueName: \"kubernetes.io/projected/a44e2389-13b8-4509-89e0-409888b5edb8-kube-api-access-qnr5n\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:01.035981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:01.035988    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a44e2389-13b8-4509-89e0-409888b5edb8-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:01.036191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:01.035996    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a44e2389-13b8-4509-89e0-409888b5edb8-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:01.830777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:01.830744    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"
Nov 03 00:28:01.880015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:01.879932    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"]
Nov 03 00:28:01.891332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:01.891302    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-27490b-7f8f4bf8f-8bg4v"]
Nov 03 00:28:02.569681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:02.569645    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44e2389-13b8-4509-89e0-409888b5edb8" path="/var/lib/kubelet/pods/a44e2389-13b8-4509-89e0-409888b5edb8/volumes"
Nov 03 00:28:03.031629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.031519    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"]
Nov 03 00:28:03.032100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.031775    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr" podUID="8c85f06b-2d34-4260-bba7-1a86a0135c9b" containerName="maas-api" containerID="cri-o://1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9" gracePeriod=30
Nov 03 00:28:03.449289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.448808    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:28:03.649479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.645998    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c85f06b-2d34-4260-bba7-1a86a0135c9b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85f06b-2d34-4260-bba7-1a86a0135c9b-openshift-service-ca\") pod \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") "
Nov 03 00:28:03.649479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.646048    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c85f06b-2d34-4260-bba7-1a86a0135c9b-kube-api-access-r2z8d\" (UniqueName: \"kubernetes.io/projected/8c85f06b-2d34-4260-bba7-1a86a0135c9b-kube-api-access-r2z8d\") pod \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") "
Nov 03 00:28:03.649479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.646101    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-maas-api-tls\") pod \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") "
Nov 03 00:28:03.649479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.646140    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-metrics-certs\") pod \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\" (UID: \"8c85f06b-2d34-4260-bba7-1a86a0135c9b\") "
Nov 03 00:28:03.653460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.653185    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c85f06b-2d34-4260-bba7-1a86a0135c9b-openshift-service-ca" pod "8c85f06b-2d34-4260-bba7-1a86a0135c9b" (UID: "8c85f06b-2d34-4260-bba7-1a86a0135c9b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:03.745407 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.744872    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-metrics-certs" pod "8c85f06b-2d34-4260-bba7-1a86a0135c9b" (UID: "8c85f06b-2d34-4260-bba7-1a86a0135c9b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:03.748416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.748358    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c85f06b-2d34-4260-bba7-1a86a0135c9b-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.748416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.748391    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.750400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.750349    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c85f06b-2d34-4260-bba7-1a86a0135c9b-kube-api-access-r2z8d" pod "8c85f06b-2d34-4260-bba7-1a86a0135c9b" (UID: "8c85f06b-2d34-4260-bba7-1a86a0135c9b"). InnerVolumeSpecName "kube-api-access-r2z8d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:03.757394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.752522    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-maas-api-tls" pod "8c85f06b-2d34-4260-bba7-1a86a0135c9b" (UID: "8c85f06b-2d34-4260-bba7-1a86a0135c9b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:03.845336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.845298    6000 generic.go:356] "Generic (PLEG): container finished" podID="8c85f06b-2d34-4260-bba7-1a86a0135c9b" containerID="1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9" exitCode=0
Nov 03 00:28:03.845535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.845347    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"
Nov 03 00:28:03.845535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.845390    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr" event={"ID":"8c85f06b-2d34-4260-bba7-1a86a0135c9b","Type":"ContainerDied","Data":"1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9"}
Nov 03 00:28:03.845535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.845447    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr" event={"ID":"8c85f06b-2d34-4260-bba7-1a86a0135c9b","Type":"ContainerDied","Data":"abd365c9c3a86d3205e46b1e20d9e4310e23974d4a32ba2974264092ba29e4ac"}
Nov 03 00:28:03.845535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.845475    6000 scope.go:122] "RemoveContainer" containerID="1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9"
Nov 03 00:28:03.849345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.849315    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2z8d\" (UniqueName: \"kubernetes.io/projected/8c85f06b-2d34-4260-bba7-1a86a0135c9b-kube-api-access-r2z8d\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.849345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.849340    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c85f06b-2d34-4260-bba7-1a86a0135c9b-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:03.855982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.855963    6000 scope.go:122] "RemoveContainer" containerID="1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9"
Nov 03 00:28:03.856237 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:28:03.856219    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9\": container with ID starting with 1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9 not found: ID does not exist" containerID="1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9"
Nov 03 00:28:03.856334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.856244    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9"} err="failed to get container status \"1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9\": rpc error: code = NotFound desc = could not find container \"1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9\": container with ID starting with 1989bc5e6959a7f53d2cc4ba267999efcf87e46757ad33eb3ca2f37af3791cb9 not found: ID does not exist"
Nov 03 00:28:03.895628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.895601    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"]
Nov 03 00:28:03.907007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:03.906952    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-d43409-8579b7fb44-kl2zr"]
Nov 03 00:28:04.570090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:04.570047    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c85f06b-2d34-4260-bba7-1a86a0135c9b" path="/var/lib/kubelet/pods/8c85f06b-2d34-4260-bba7-1a86a0135c9b/volumes"
Nov 03 00:28:07.764701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.764662    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"]
Nov 03 00:28:07.765542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765516    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" containerName="payload-pre-processing"
Nov 03 00:28:07.765542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765541    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" containerName="payload-pre-processing"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765566    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7014038a-0215-4164-b032-feff0b68056f" containerName="maas-api"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765573    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7014038a-0215-4164-b032-feff0b68056f" containerName="maas-api"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765598    6000 cpu_manager.go:414] "containerMap: removing container" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765606    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765617    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f4609dd8-273d-43a9-94ac-70e51d339fa2" containerName="maas-api"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765623    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4609dd8-273d-43a9-94ac-70e51d339fa2" containerName="maas-api"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765641    6000 cpu_manager.go:414] "containerMap: removing container" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765648    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765661    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765668    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765687    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765694    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b6fc11-a279-4997-a8bf-c196b8650d8c" containerName="istio-proxy"
Nov 03 00:28:07.765710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765713    6000 cpu_manager.go:414] "containerMap: removing container" podUID="a6765be6-efe8-457f-be3d-3e2fd547684d" containerName="payload-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765721    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6765be6-efe8-457f-be3d-3e2fd547684d" containerName="payload-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765735    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8c85f06b-2d34-4260-bba7-1a86a0135c9b" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765741    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c85f06b-2d34-4260-bba7-1a86a0135c9b" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765756    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" containerName="payload-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765763    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" containerName="payload-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765778    6000 cpu_manager.go:414] "containerMap: removing container" podUID="aa0de994-cdee-4f27-b8ca-860b8e31b169" containerName="payload-pre-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765785    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa0de994-cdee-4f27-b8ca-860b8e31b169" containerName="payload-pre-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765805    6000 cpu_manager.go:414] "containerMap: removing container" podUID="a44e2389-13b8-4509-89e0-409888b5edb8" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765812    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44e2389-13b8-4509-89e0-409888b5edb8" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765976    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7014038a-0215-4164-b032-feff0b68056f" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.765990    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="529a05c4-4cd7-4729-bcb9-70cfa9b6d448" containerName="istio-proxy"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766003    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4609dd8-273d-43a9-94ac-70e51d339fa2" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766016    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="35ceba1e-6211-4524-9062-037a85917c22" containerName="istio-proxy"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766028    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d22b28f-afb0-4b8c-96a8-31e7cdb63c50" containerName="istio-proxy"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766038    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6765be6-efe8-457f-be3d-3e2fd547684d" containerName="payload-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766046    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c85f06b-2d34-4260-bba7-1a86a0135c9b" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766054    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f516bf3a-d7a5-4653-80f3-f17fb2ed9b42" containerName="payload-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766065    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa0de994-cdee-4f27-b8ca-860b8e31b169" containerName="payload-pre-processing"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766078    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44e2389-13b8-4509-89e0-409888b5edb8" containerName="maas-api"
Nov 03 00:28:07.766668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.766090    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aba12fe-31b5-4eba-b3be-fdccd6b3cedf" containerName="payload-pre-processing"
Nov 03 00:28:07.771529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.771508    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.788604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.788572    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"]
Nov 03 00:28:07.796234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.796212    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e37c9d-serving-cert\""
Nov 03 00:28:07.829177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.829139    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c48858-5f75-4623-87d7-557a220968a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.829400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.829354    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.829559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.829528    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.829694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.829592    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/05c48858-5f75-4623-87d7-557a220968a1-kube-api-access-b5f6r\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.930539 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.930505    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c48858-5f75-4623-87d7-557a220968a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.930722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.930600    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.930824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.930761    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.930824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.930787    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/05c48858-5f75-4623-87d7-557a220968a1-kube-api-access-b5f6r\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.931328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.931301    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c48858-5f75-4623-87d7-557a220968a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.933192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.933168    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.933346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.933320    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:07.949183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:07.949158    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/05c48858-5f75-4623-87d7-557a220968a1-kube-api-access-b5f6r\") pod \"maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:08.090336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:08.086202    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:08.090336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:08.088112    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:28:08.446828 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:28:08.446796    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05c48858_5f75_4623_87d7_557a220968a1.slice/crio-84affccb18f1f5394b8abbee438fb17a61f62d29d57ab1e9fed3e164711832fe WatchSource:0}: Error finding container 84affccb18f1f5394b8abbee438fb17a61f62d29d57ab1e9fed3e164711832fe: Status 404 returned error can't find the container with id 84affccb18f1f5394b8abbee438fb17a61f62d29d57ab1e9fed3e164711832fe
Nov 03 00:28:08.447751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:08.447726    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"]
Nov 03 00:28:09.113387 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:09.113311    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4" event={"ID":"05c48858-5f75-4623-87d7-557a220968a1","Type":"ContainerStarted","Data":"84affccb18f1f5394b8abbee438fb17a61f62d29d57ab1e9fed3e164711832fe"}
Nov 03 00:28:10.134011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:10.133968    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4" event={"ID":"05c48858-5f75-4623-87d7-557a220968a1","Type":"ContainerStarted","Data":"a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298"}
Nov 03 00:28:10.134641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:10.134213    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:10.177910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:10.177846    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4" podStartSLOduration=2.753763314 podStartE2EDuration="3.177830813s" totalImagesPullingTime="424.067499ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:10.174221726 +0000 UTC m=+1327.729573766" watchObservedRunningTime="2026-11-03 00:28:10.177830813 +0000 UTC m=+1327.733182836"
Nov 03 00:28:13.025194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.025161    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"]
Nov 03 00:28:13.029887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.029863    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.052968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.052940    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-54f6d8-openshift-default-dockercfg-4swp6\""
Nov 03 00:28:13.067385 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.067346    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"]
Nov 03 00:28:13.224224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.221241    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-certs\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.224884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.224853    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-podinfo\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.225127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.225112    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istiod-ca-cert\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.225234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.225223    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-socket\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.225382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.225366    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-token\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.246512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.246470    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-data\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.270336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.270300    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbdgd\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-kube-api-access-kbdgd\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.270630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.270607    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-envoy\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.270813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.270796    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-credential-socket\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.378404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.378366    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-certs\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.388587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.379194    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-podinfo\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.389036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.379076    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-certs\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.389305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.389031    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istiod-ca-cert\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.389489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.389464    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-socket\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.389620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.389602    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-token\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.389839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.389824    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-data\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.390121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.390103    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbdgd\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-kube-api-access-kbdgd\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.390285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.390248    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-envoy\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.390481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.390465    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-credential-socket\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.390995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.390971    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-socket\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.391685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.391571    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-credential-socket\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.392086 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.392070    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-data\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.392302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.390355    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istiod-ca-cert\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.435819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.433429    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-envoy\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.437075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.436768    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbdgd\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-kube-api-access-kbdgd\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.440725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.440703    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-token\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.441749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.441729    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-podinfo\") pod \"e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") " pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.643495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.643410    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:13.957222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.957184    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"]
Nov 03 00:28:13.957222 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:28:13.957211    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef1b4163_a397_484b_973f_a4d8fb9f6e88.slice/crio-36d1309f6cf72e05a5163a6f506d991b81ce31ce6b18fa0a26a04415d7e1ddec WatchSource:0}: Error finding container 36d1309f6cf72e05a5163a6f506d991b81ce31ce6b18fa0a26a04415d7e1ddec: Status 404 returned error can't find the container with id 36d1309f6cf72e05a5163a6f506d991b81ce31ce6b18fa0a26a04415d7e1ddec
Nov 03 00:28:13.959947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.959915    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:28:13.960037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.959979    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:28:13.960037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:13.960009    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:28:14.397222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:14.395735    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" event={"ID":"ef1b4163-a397-484b-973f-a4d8fb9f6e88","Type":"ContainerStarted","Data":"752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97"}
Nov 03 00:28:14.397222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:14.395806    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" event={"ID":"ef1b4163-a397-484b-973f-a4d8fb9f6e88","Type":"ContainerStarted","Data":"36d1309f6cf72e05a5163a6f506d991b81ce31ce6b18fa0a26a04415d7e1ddec"}
Nov 03 00:28:14.444275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:14.443628    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" podStartSLOduration=2.443601461 podStartE2EDuration="2.443601461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:14.440686849 +0000 UTC m=+1331.996038884" watchObservedRunningTime="2026-11-03 00:28:14.443601461 +0000 UTC m=+1331.998953506"
Nov 03 00:28:14.644344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:14.643961    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:15.649629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:15.649600    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:16.145908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:16.145881    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:28:16.408327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:16.408210    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:16.409218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:16.409198    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:28:19.977547 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:28:19.977480    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:28:26.104624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.104590    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"]
Nov 03 00:28:26.105147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.104805    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn" podUID="fad28b7c-811a-4225-a1b5-de0a3138978e" containerName="maas-api" containerID="cri-o://2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2" gracePeriod=30
Nov 03 00:28:26.172455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.172414    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn" podUID="fad28b7c-811a-4225-a1b5-de0a3138978e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.104:8443/health\": dial tcp 10.134.0.104:8443: connect: connection refused"
Nov 03 00:28:26.402749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.402722    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:28:26.418919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.418882    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-metrics-certs\") pod \"fad28b7c-811a-4225-a1b5-de0a3138978e\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") "
Nov 03 00:28:26.419076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.419028    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fad28b7c-811a-4225-a1b5-de0a3138978e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad28b7c-811a-4225-a1b5-de0a3138978e-openshift-service-ca\") pod \"fad28b7c-811a-4225-a1b5-de0a3138978e\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") "
Nov 03 00:28:26.419176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.419163    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-maas-api-tls\") pod \"fad28b7c-811a-4225-a1b5-de0a3138978e\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") "
Nov 03 00:28:26.419334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.419246    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fad28b7c-811a-4225-a1b5-de0a3138978e-kube-api-access-nrpdx\" (UniqueName: \"kubernetes.io/projected/fad28b7c-811a-4225-a1b5-de0a3138978e-kube-api-access-nrpdx\") pod \"fad28b7c-811a-4225-a1b5-de0a3138978e\" (UID: \"fad28b7c-811a-4225-a1b5-de0a3138978e\") "
Nov 03 00:28:26.419441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.419425    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad28b7c-811a-4225-a1b5-de0a3138978e-openshift-service-ca" pod "fad28b7c-811a-4225-a1b5-de0a3138978e" (UID: "fad28b7c-811a-4225-a1b5-de0a3138978e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:26.419887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.419867    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad28b7c-811a-4225-a1b5-de0a3138978e-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:26.421195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.421168    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-metrics-certs" pod "fad28b7c-811a-4225-a1b5-de0a3138978e" (UID: "fad28b7c-811a-4225-a1b5-de0a3138978e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:26.421482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.421458    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fad28b7c-811a-4225-a1b5-de0a3138978e-kube-api-access-nrpdx" pod "fad28b7c-811a-4225-a1b5-de0a3138978e" (UID: "fad28b7c-811a-4225-a1b5-de0a3138978e"). InnerVolumeSpecName "kube-api-access-nrpdx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:26.421571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.421495    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-maas-api-tls" pod "fad28b7c-811a-4225-a1b5-de0a3138978e" (UID: "fad28b7c-811a-4225-a1b5-de0a3138978e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:26.521189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.521157    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrpdx\" (UniqueName: \"kubernetes.io/projected/fad28b7c-811a-4225-a1b5-de0a3138978e-kube-api-access-nrpdx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:26.521189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.521188    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:26.521189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.521196    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fad28b7c-811a-4225-a1b5-de0a3138978e-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:26.525177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.525139    6000 generic.go:356] "Generic (PLEG): container finished" podID="fad28b7c-811a-4225-a1b5-de0a3138978e" containerID="2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2" exitCode=0
Nov 03 00:28:26.525348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.525191    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn" event={"ID":"fad28b7c-811a-4225-a1b5-de0a3138978e","Type":"ContainerDied","Data":"2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2"}
Nov 03 00:28:26.525348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.525275    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn" event={"ID":"fad28b7c-811a-4225-a1b5-de0a3138978e","Type":"ContainerDied","Data":"a7fa93cc40639fe7eda167be898900333688659f6d0e86bb387490a88e814f8a"}
Nov 03 00:28:26.525348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.525203    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"
Nov 03 00:28:26.525348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.525305    6000 scope.go:122] "RemoveContainer" containerID="2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2"
Nov 03 00:28:26.538396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.538376    6000 scope.go:122] "RemoveContainer" containerID="2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2"
Nov 03 00:28:26.538686 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:28:26.538670    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2\": container with ID starting with 2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2 not found: ID does not exist" containerID="2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2"
Nov 03 00:28:26.538752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.538700    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2"} err="failed to get container status \"2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2\": rpc error: code = NotFound desc = could not find container \"2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2\": container with ID starting with 2aeace35c259142ced893ca229ff6fe74191dbedf04c11e18b8c5c559d5b46f2 not found: ID does not exist"
Nov 03 00:28:26.733756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.733671    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"]
Nov 03 00:28:26.780283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:26.768037    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-41e815-9667d94cd-rpbcn"]
Nov 03 00:28:28.567233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:28.567199    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fad28b7c-811a-4225-a1b5-de0a3138978e" path="/var/lib/kubelet/pods/fad28b7c-811a-4225-a1b5-de0a3138978e/volumes"
Nov 03 00:28:30.122999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:30.122967    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"]
Nov 03 00:28:30.123642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:30.123279    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy" containerID="cri-o://c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810" gracePeriod=30
Nov 03 00:28:34.371805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:34.371772    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:28:34.372270 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:34.371832    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:28:35.404517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.404453    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:28:35.488099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488065    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93aecdc9-6974-42f6-b72d-c6da546c6c91-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aecdc9-6974-42f6-b72d-c6da546c6c91-istiod-ca-cert\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488143    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-kube-api-access-nq45k\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-kube-api-access-nq45k\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488174    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-credential-socket\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488192    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-data\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488271    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-envoy\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488307    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-certs\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488420    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-podinfo\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488542    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-socket\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488618    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-token\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-token\") pod \"93aecdc9-6974-42f6-b72d-c6da546c6c91\" (UID: \"93aecdc9-6974-42f6-b72d-c6da546c6c91\") "
Nov 03 00:28:35.488930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.488693    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-credential-socket" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:35.489560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.489040    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93aecdc9-6974-42f6-b72d-c6da546c6c91-istiod-ca-cert" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:35.489560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.489119    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.489560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.489247    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-certs" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:35.489560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.489299    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-data" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:35.489560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.489526    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-socket" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:35.492141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.492105    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-podinfo" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:28:35.492780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.492757    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-token" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:35.493029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.493010    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-kube-api-access-nq45k" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "kube-api-access-nq45k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:35.499554 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.499527    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-envoy" pod "93aecdc9-6974-42f6-b72d-c6da546c6c91" (UID: "93aecdc9-6974-42f6-b72d-c6da546c6c91"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593074    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593114    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593128    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/93aecdc9-6974-42f6-b72d-c6da546c6c91-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593140    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq45k\" (UniqueName: \"kubernetes.io/projected/93aecdc9-6974-42f6-b72d-c6da546c6c91-kube-api-access-nq45k\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593155    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593167    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593178    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/93aecdc9-6974-42f6-b72d-c6da546c6c91-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.594135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.593192    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/93aecdc9-6974-42f6-b72d-c6da546c6c91-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:35.630629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.630590    6000 generic.go:356] "Generic (PLEG): container finished" podID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerID="c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810" exitCode=0
Nov 03 00:28:35.630813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.630698    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"
Nov 03 00:28:35.630813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.630711    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" event={"ID":"93aecdc9-6974-42f6-b72d-c6da546c6c91","Type":"ContainerDied","Data":"c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810"}
Nov 03 00:28:35.630813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.630767    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z" event={"ID":"93aecdc9-6974-42f6-b72d-c6da546c6c91","Type":"ContainerDied","Data":"8d70e0cdf9096f31d726618a869cc75bf4904c109f81671c2a695dc18b1a7559"}
Nov 03 00:28:35.630813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.630795    6000 scope.go:122] "RemoveContainer" containerID="c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810"
Nov 03 00:28:35.642042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.642023    6000 scope.go:122] "RemoveContainer" containerID="8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b"
Nov 03 00:28:35.652393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.652374    6000 scope.go:122] "RemoveContainer" containerID="c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810"
Nov 03 00:28:35.652689 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:28:35.652668    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810\": container with ID starting with c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810 not found: ID does not exist" containerID="c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810"
Nov 03 00:28:35.652763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.652696    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810"} err="failed to get container status \"c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810\": rpc error: code = NotFound desc = could not find container \"c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810\": container with ID starting with c2e87dde67020d64c82c0f843b35ee60a639d73d86b56edf91f2ba5dbe3fa810 not found: ID does not exist"
Nov 03 00:28:35.652763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.652713    6000 scope.go:122] "RemoveContainer" containerID="8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b"
Nov 03 00:28:35.652997 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:28:35.652977    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b\": container with ID starting with 8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b not found: ID does not exist" containerID="8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b"
Nov 03 00:28:35.653087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.652999    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b"} err="failed to get container status \"8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b\": rpc error: code = NotFound desc = could not find container \"8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b\": container with ID starting with 8cca8b0e9b9f104f775054bd16fb35961d8a08b96b2cfc9ad04cf31764fc717b not found: ID does not exist"
Nov 03 00:28:35.680751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.680686    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"]
Nov 03 00:28:35.683326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:35.683304    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-41e815-openshift-default-649b9cbc9-cpn5z"]
Nov 03 00:28:36.615320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:36.608999    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" path="/var/lib/kubelet/pods/93aecdc9-6974-42f6-b72d-c6da546c6c91/volumes"
Nov 03 00:28:37.806750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.806718    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"]
Nov 03 00:28:37.807544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807524    6000 cpu_manager.go:414] "containerMap: removing container" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy"
Nov 03 00:28:37.807609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807593    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy"
Nov 03 00:28:37.807653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807614    6000 cpu_manager.go:414] "containerMap: removing container" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy"
Nov 03 00:28:37.807653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807620    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy"
Nov 03 00:28:37.807653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807635    6000 cpu_manager.go:414] "containerMap: removing container" podUID="fad28b7c-811a-4225-a1b5-de0a3138978e" containerName="maas-api"
Nov 03 00:28:37.807653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807641    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fad28b7c-811a-4225-a1b5-de0a3138978e" containerName="maas-api"
Nov 03 00:28:37.807770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807762    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy"
Nov 03 00:28:37.807803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807771    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="93aecdc9-6974-42f6-b72d-c6da546c6c91" containerName="istio-proxy"
Nov 03 00:28:37.807803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.807779    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="fad28b7c-811a-4225-a1b5-de0a3138978e" containerName="maas-api"
Nov 03 00:28:37.812836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.812819    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:37.830244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.830212    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"]
Nov 03 00:28:37.846300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.837754    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-54f6d8-serving-cert\""
Nov 03 00:28:37.955363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.955311    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:37.955573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.955499    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:37.955652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.955632    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:37.955752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:37.955737    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdbst\" (UniqueName: \"kubernetes.io/projected/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-kube-api-access-jdbst\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.065328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.060918    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.065328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.061140    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.065328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.061218    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.065328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.061344    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdbst\" (UniqueName: \"kubernetes.io/projected/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-kube-api-access-jdbst\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.081281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.076997    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.123317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.114552    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdbst\" (UniqueName: \"kubernetes.io/projected/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-kube-api-access-jdbst\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.123317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.120156    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-metrics-certs\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.123317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.120730    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-maas-api-tls\") pod \"maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.133277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.130513    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:38.499589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.492975    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"]
Nov 03 00:28:38.680550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:38.680514    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx" event={"ID":"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe","Type":"ContainerStarted","Data":"682af6a36b33726cd8f99cac6e195d70be0aba72fa611530e9340ccee5724b5e"}
Nov 03 00:28:39.688889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:39.688856    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx" event={"ID":"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe","Type":"ContainerStarted","Data":"7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1"}
Nov 03 00:28:39.689365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:39.688933    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:39.726851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:39.726790    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx" podStartSLOduration=2.4244638050000002 podStartE2EDuration="2.726772553s" totalImagesPullingTime="302.308748ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:39.725714857 +0000 UTC m=+1357.281066882" watchObservedRunningTime="2026-11-03 00:28:39.726772553 +0000 UTC m=+1357.282124578"
Nov 03 00:28:42.605241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.605143    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"]
Nov 03 00:28:42.612047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.612022    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:42.632717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.632687    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"]
Nov 03 00:28:42.637518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.637490    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-846492-dockercfg-lnjvr\""
Nov 03 00:28:42.675645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.675422    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-846492\""
Nov 03 00:28:42.756283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.747953    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cde5e87-df70-4e15-8e36-bf7587b15550-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:42.756283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.748028    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gfnkq\" (UniqueName: \"kubernetes.io/projected/8cde5e87-df70-4e15-8e36-bf7587b15550-kube-api-access-gfnkq\") pod \"payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:42.867332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.866456    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cde5e87-df70-4e15-8e36-bf7587b15550-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:42.867332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.866540    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gfnkq\" (UniqueName: \"kubernetes.io/projected/8cde5e87-df70-4e15-8e36-bf7587b15550-kube-api-access-gfnkq\") pod \"payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:42.879556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.868277    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cde5e87-df70-4e15-8e36-bf7587b15550-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:42.926279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.925911    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gfnkq\" (UniqueName: \"kubernetes.io/projected/8cde5e87-df70-4e15-8e36-bf7587b15550-kube-api-access-gfnkq\") pod \"payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:42.953994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:42.953960    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:43.267921 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:28:43.267888    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cde5e87_df70_4e15_8e36_bf7587b15550.slice/crio-163ba2d650c7c81bfd8c49b782abf309158b79535038bbc72ea87ef4adecf8f1 WatchSource:0}: Error finding container 163ba2d650c7c81bfd8c49b782abf309158b79535038bbc72ea87ef4adecf8f1: Status 404 returned error can't find the container with id 163ba2d650c7c81bfd8c49b782abf309158b79535038bbc72ea87ef4adecf8f1
Nov 03 00:28:43.268822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:43.268791    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"]
Nov 03 00:28:43.920011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:43.919970    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4" event={"ID":"8cde5e87-df70-4e15-8e36-bf7587b15550","Type":"ContainerStarted","Data":"930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790"}
Nov 03 00:28:43.920543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:43.920065    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4" event={"ID":"8cde5e87-df70-4e15-8e36-bf7587b15550","Type":"ContainerStarted","Data":"163ba2d650c7c81bfd8c49b782abf309158b79535038bbc72ea87ef4adecf8f1"}
Nov 03 00:28:43.921076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:43.921050    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:43.963388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:43.963338    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4" podStartSLOduration=1.96332283 podStartE2EDuration="1.96332283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:28:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:28:43.96041303 +0000 UTC m=+1361.515765054" watchObservedRunningTime="2026-11-03 00:28:43.96332283 +0000 UTC m=+1361.518674854"
Nov 03 00:28:45.702848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:45.702820    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:28:47.517414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.517383    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"]
Nov 03 00:28:47.517806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.517564    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5" podUID="7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" containerName="maas-api" containerID="cri-o://fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235" gracePeriod=30
Nov 03 00:28:47.822580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.822553    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:28:47.858229 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.858201    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-maas-api-tls\") pod \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") "
Nov 03 00:28:47.858437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.858423    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-metrics-certs\") pod \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") "
Nov 03 00:28:47.858508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.858456    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-openshift-service-ca\") pod \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") "
Nov 03 00:28:47.858569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.858511    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-kube-api-access-cgbxm\" (UniqueName: \"kubernetes.io/projected/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-kube-api-access-cgbxm\") pod \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\" (UID: \"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427\") "
Nov 03 00:28:47.858886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.858854    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-openshift-service-ca" pod "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" (UID: "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:28:47.859068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.859048    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:47.861006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.860981    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-maas-api-tls" pod "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" (UID: "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:47.861732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.861704    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-metrics-certs" pod "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" (UID: "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:28:47.861959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.861904    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-kube-api-access-cgbxm" pod "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" (UID: "7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427"). InnerVolumeSpecName "kube-api-access-cgbxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:28:47.960857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.960827    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:47.960857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.960854    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgbxm\" (UniqueName: \"kubernetes.io/projected/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-kube-api-access-cgbxm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:47.960857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.960864    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:28:47.982194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.982165    6000 generic.go:356] "Generic (PLEG): container finished" podID="7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" containerID="fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235" exitCode=0
Nov 03 00:28:47.982362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.982211    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"
Nov 03 00:28:47.982362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.982237    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5" event={"ID":"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427","Type":"ContainerDied","Data":"fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235"}
Nov 03 00:28:47.982362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.982314    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5" event={"ID":"7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427","Type":"ContainerDied","Data":"7bc65c1b9a16ff4aaf9ccf07a43de5b1951dbe7ebc6e8cbc3962719058e5fe3e"}
Nov 03 00:28:47.982362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.982337    6000 scope.go:122] "RemoveContainer" containerID="fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235"
Nov 03 00:28:47.993506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.993490    6000 scope.go:122] "RemoveContainer" containerID="fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235"
Nov 03 00:28:47.993793 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:28:47.993772    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235\": container with ID starting with fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235 not found: ID does not exist" containerID="fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235"
Nov 03 00:28:47.993838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:47.993803    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235"} err="failed to get container status \"fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235\": rpc error: code = NotFound desc = could not find container \"fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235\": container with ID starting with fe2d1419d6511af8538caa515ff11e31296a9bc300c6da917e367ce0370e1235 not found: ID does not exist"
Nov 03 00:28:48.131280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:48.110844    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"]
Nov 03 00:28:48.131280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:48.122887    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-dda6df-7774bb4dc6-m2rm5"]
Nov 03 00:28:48.567862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:48.567787    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" path="/var/lib/kubelet/pods/7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427/volumes"
Nov 03 00:28:49.901390 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:28:49.901352    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:28:55.973017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:55.972985    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:28:56.562305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:28:56.561325    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:29:00.529957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.529900    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"]
Nov 03 00:29:00.530740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.530723    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" containerName="maas-api"
Nov 03 00:29:00.530740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.530741    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" containerName="maas-api"
Nov 03 00:29:00.530901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.530882    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c0d75f1-9ccf-443b-adfa-6d4f9ff7c427" containerName="maas-api"
Nov 03 00:29:00.534779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.534758    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.550432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.550410    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-846492-serving-cert\""
Nov 03 00:29:00.572740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.572698    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"]
Nov 03 00:29:00.664208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.664177    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z98qj\" (UniqueName: \"kubernetes.io/projected/1f1ddfe4-135a-4efc-af63-ba784dabca6a-kube-api-access-z98qj\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.664436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.664314    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-metrics-certs\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.664543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.664521    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.664609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.664575    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1ddfe4-135a-4efc-af63-ba784dabca6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.765643 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.765611    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-metrics-certs\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.765842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.765700    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.765842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.765728    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1ddfe4-135a-4efc-af63-ba784dabca6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.765842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.765815    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z98qj\" (UniqueName: \"kubernetes.io/projected/1f1ddfe4-135a-4efc-af63-ba784dabca6a-kube-api-access-z98qj\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.766461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.766436    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1ddfe4-135a-4efc-af63-ba784dabca6a-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.768357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.768332    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-maas-api-tls\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.768505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.768482    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-metrics-certs\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.845330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.834370    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z98qj\" (UniqueName: \"kubernetes.io/projected/1f1ddfe4-135a-4efc-af63-ba784dabca6a-kube-api-access-z98qj\") pod \"maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:00.864280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:00.854547    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:01.243637 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:01.243599    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f1ddfe4_135a_4efc_af63_ba784dabca6a.slice/crio-22dbac9e60435e56db55364d97aee18699883340ffe240f3de14323f9bfc4cc4 WatchSource:0}: Error finding container 22dbac9e60435e56db55364d97aee18699883340ffe240f3de14323f9bfc4cc4: Status 404 returned error can't find the container with id 22dbac9e60435e56db55364d97aee18699883340ffe240f3de14323f9bfc4cc4
Nov 03 00:29:01.243761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:01.243633    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"]
Nov 03 00:29:02.261300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.261236    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf" event={"ID":"1f1ddfe4-135a-4efc-af63-ba784dabca6a","Type":"ContainerStarted","Data":"fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b"}
Nov 03 00:29:02.261300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.261302    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf" event={"ID":"1f1ddfe4-135a-4efc-af63-ba784dabca6a","Type":"ContainerStarted","Data":"22dbac9e60435e56db55364d97aee18699883340ffe240f3de14323f9bfc4cc4"}
Nov 03 00:29:02.261753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.261344    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:02.542200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.542101    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf" podStartSLOduration=2.067984905 podStartE2EDuration="2.542087351s" totalImagesPullingTime="474.102446ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:02.320480484 +0000 UTC m=+1379.875832511" watchObservedRunningTime="2026-11-03 00:29:02.542087351 +0000 UTC m=+1380.097439388"
Nov 03 00:29:02.544391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.544364    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"]
Nov 03 00:29:02.548932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.548912    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:02.566566 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.566227    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4ae722-dockercfg-8wrbq\""
Nov 03 00:29:02.571611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.571585    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"]
Nov 03 00:29:02.643321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.623572    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-4ae722\""
Nov 03 00:29:02.955279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.952592    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr5dw\" (UniqueName: \"kubernetes.io/projected/65cf5dee-fcb9-4598-ada6-ff8209a638a8-kube-api-access-fr5dw\") pod \"payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:02.955279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:02.952791    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cf5dee-fcb9-4598-ada6-ff8209a638a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:03.054857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:03.054819    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr5dw\" (UniqueName: \"kubernetes.io/projected/65cf5dee-fcb9-4598-ada6-ff8209a638a8-kube-api-access-fr5dw\") pod \"payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:03.055055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:03.054956    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cf5dee-fcb9-4598-ada6-ff8209a638a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:03.055876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:03.055847    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cf5dee-fcb9-4598-ada6-ff8209a638a8-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:03.077200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:03.077167    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr5dw\" (UniqueName: \"kubernetes.io/projected/65cf5dee-fcb9-4598-ada6-ff8209a638a8-kube-api-access-fr5dw\") pod \"payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:03.190989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:03.180718    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-4ae722-dockercfg-8wrbq\""
Nov 03 00:29:03.190989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:03.188504    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:03.678276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:03.675940    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"]
Nov 03 00:29:04.336143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:04.336106    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" event={"ID":"65cf5dee-fcb9-4598-ada6-ff8209a638a8","Type":"ContainerStarted","Data":"4bbab479da1f99417c6b4bf313fad2bc72988f599c5bfbea092cada2044c264d"}
Nov 03 00:29:04.336360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:04.336188    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" event={"ID":"65cf5dee-fcb9-4598-ada6-ff8209a638a8","Type":"ContainerStarted","Data":"38b3c9d5648ed7d471e561cf21abc62639a9fb62ebc078ad04edb3757dca2634"}
Nov 03 00:29:04.336360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:04.336211    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:04.376541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:04.376475    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" podStartSLOduration=2.376455749 podStartE2EDuration="2.376455749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:04.371789241 +0000 UTC m=+1381.927141291" watchObservedRunningTime="2026-11-03 00:29:04.376455749 +0000 UTC m=+1381.931807774"
Nov 03 00:29:08.334621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:08.334540    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:29:08.411709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:08.411623    6000 generic.go:356] "Generic (PLEG): container finished" podID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerID="752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97" exitCode=0
Nov 03 00:29:08.411709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:08.411680    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" event={"ID":"ef1b4163-a397-484b-973f-a4d8fb9f6e88","Type":"ContainerDied","Data":"752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97"}
Nov 03 00:29:08.412194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:08.412174    6000 scope.go:122] "RemoveContainer" containerID="752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97"
Nov 03 00:29:08.415471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:08.415431    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:08.415577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:08.415512    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:08.415577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:08.415562    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:09.419867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:09.419820    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" event={"ID":"ef1b4163-a397-484b-973f-a4d8fb9f6e88","Type":"ContainerStarted","Data":"dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357"}
Nov 03 00:29:09.646001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:09.645968    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:29:09.646300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:09.646081    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:29:09.646450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:09.646429    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 00:29:10.644558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:10.644520    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:29:10.645061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:10.644580    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 00:29:11.643715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:11.643678    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:29:11.643903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:11.643733    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 00:29:12.644422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:12.644382    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:29:12.644843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:12.644435    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 00:29:13.644420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:13.644390    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body=
Nov 03 00:29:13.644611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:13.644435    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused"
Nov 03 00:29:14.647908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:14.647878    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:29:14.648443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:14.648279    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:29:14.649039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:14.649018    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:29:15.430686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:15.430654    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:29:18.594102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.594067    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"]
Nov 03 00:29:18.594714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.594329    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6" podUID="c1c25b44-8d36-473c-9878-fbafc221423c" containerName="maas-api" containerID="cri-o://1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58" gracePeriod=30
Nov 03 00:29:18.887579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.887552    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:29:18.960767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.960729    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls\") pod \"c1c25b44-8d36-473c-9878-fbafc221423c\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") "
Nov 03 00:29:18.960966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.960818    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c25b44-8d36-473c-9878-fbafc221423c-kube-api-access-h89t2\" (UniqueName: \"kubernetes.io/projected/c1c25b44-8d36-473c-9878-fbafc221423c-kube-api-access-h89t2\") pod \"c1c25b44-8d36-473c-9878-fbafc221423c\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") "
Nov 03 00:29:18.960966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.960864    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-metrics-certs\") pod \"c1c25b44-8d36-473c-9878-fbafc221423c\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") "
Nov 03 00:29:18.961189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.961166    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c25b44-8d36-473c-9878-fbafc221423c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c25b44-8d36-473c-9878-fbafc221423c-openshift-service-ca\") pod \"c1c25b44-8d36-473c-9878-fbafc221423c\" (UID: \"c1c25b44-8d36-473c-9878-fbafc221423c\") "
Nov 03 00:29:18.961569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.961546    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c25b44-8d36-473c-9878-fbafc221423c-openshift-service-ca" pod "c1c25b44-8d36-473c-9878-fbafc221423c" (UID: "c1c25b44-8d36-473c-9878-fbafc221423c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:18.961921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.961888    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1c25b44-8d36-473c-9878-fbafc221423c-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:18.964364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.964340    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c25b44-8d36-473c-9878-fbafc221423c-kube-api-access-h89t2" pod "c1c25b44-8d36-473c-9878-fbafc221423c" (UID: "c1c25b44-8d36-473c-9878-fbafc221423c"). InnerVolumeSpecName "kube-api-access-h89t2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:18.964608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.964585    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls" pod "c1c25b44-8d36-473c-9878-fbafc221423c" (UID: "c1c25b44-8d36-473c-9878-fbafc221423c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:29:18.966364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:18.966337    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-metrics-certs" pod "c1c25b44-8d36-473c-9878-fbafc221423c" (UID: "c1c25b44-8d36-473c-9878-fbafc221423c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:29:19.098970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.071629    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:19.109470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.109392    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h89t2\" (UniqueName: \"kubernetes.io/projected/c1c25b44-8d36-473c-9878-fbafc221423c-kube-api-access-h89t2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:19.109698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.109677    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c1c25b44-8d36-473c-9878-fbafc221423c-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:19.519052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.518968    6000 generic.go:356] "Generic (PLEG): container finished" podID="c1c25b44-8d36-473c-9878-fbafc221423c" containerID="1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58" exitCode=0
Nov 03 00:29:19.519052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.519019    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6" event={"ID":"c1c25b44-8d36-473c-9878-fbafc221423c","Type":"ContainerDied","Data":"1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58"}
Nov 03 00:29:19.519052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.519036    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"
Nov 03 00:29:19.519402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.519061    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6" event={"ID":"c1c25b44-8d36-473c-9878-fbafc221423c","Type":"ContainerDied","Data":"923fd1ac6f88cb8cac9572db73956accad5bd2be1512ea4e8ffd22b967be6272"}
Nov 03 00:29:19.519402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.519081    6000 scope.go:122] "RemoveContainer" containerID="1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58"
Nov 03 00:29:19.530213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.530196    6000 scope.go:122] "RemoveContainer" containerID="1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58"
Nov 03 00:29:19.530548 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:29:19.530528    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58\": container with ID starting with 1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58 not found: ID does not exist" containerID="1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58"
Nov 03 00:29:19.530651 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.530557    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58"} err="failed to get container status \"1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58\": rpc error: code = NotFound desc = could not find container \"1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58\": container with ID starting with 1c083e639b825122950d371af018c783f6cd861c329dfca8f29e6601a0af3c58 not found: ID does not exist"
Nov 03 00:29:19.594448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.594421    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"]
Nov 03 00:29:19.609690 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:19.609667    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-3437aa-8b9fd8f7c-rt2m6"]
Nov 03 00:29:19.898945 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:29:19.898904    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:29:20.567612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:20.567578    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c25b44-8d36-473c-9878-fbafc221423c" path="/var/lib/kubelet/pods/c1c25b44-8d36-473c-9878-fbafc221423c/volumes"
Nov 03 00:29:22.218281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.199569    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"]
Nov 03 00:29:22.218281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.200518    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c1c25b44-8d36-473c-9878-fbafc221423c" containerName="maas-api"
Nov 03 00:29:22.218281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.200536    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c25b44-8d36-473c-9878-fbafc221423c" containerName="maas-api"
Nov 03 00:29:22.218281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.200668    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c25b44-8d36-473c-9878-fbafc221423c" containerName="maas-api"
Nov 03 00:29:22.218955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.218566    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.228594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.228494    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"]
Nov 03 00:29:22.269126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.269087    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-4ae722-serving-cert\""
Nov 03 00:29:22.327718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.327679    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b3b3d1-a353-4a58-aadb-0e845ed8e755-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.327889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.327851    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.327931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.327914    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-metrics-certs\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.328026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.328012    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvcx\" (UniqueName: \"kubernetes.io/projected/67b3b3d1-a353-4a58-aadb-0e845ed8e755-kube-api-access-8xvcx\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.437152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.437103    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.445941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.445900    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-metrics-certs\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.446158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.446039    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvcx\" (UniqueName: \"kubernetes.io/projected/67b3b3d1-a353-4a58-aadb-0e845ed8e755-kube-api-access-8xvcx\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.446228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.446209    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b3b3d1-a353-4a58-aadb-0e845ed8e755-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.454614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.447145    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b3b3d1-a353-4a58-aadb-0e845ed8e755-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.483114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.483025    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-metrics-certs\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.512967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.512912    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-maas-api-tls\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.513786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.513757    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvcx\" (UniqueName: \"kubernetes.io/projected/67b3b3d1-a353-4a58-aadb-0e845ed8e755-kube-api-access-8xvcx\") pod \"maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.544361 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.544328    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:22.650328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.650287    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"]
Nov 03 00:29:22.652229 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.652193    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" containerID="cri-o://546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5" gracePeriod=30
Nov 03 00:29:22.974882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.974848    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"]
Nov 03 00:29:22.977306 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:22.976020    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b3b3d1_a353_4a58_aadb_0e845ed8e755.slice/crio-ca82481e2b0c7a085a715e23580a7ec47f1c53a3dee3850ac881cf0e1c95eb66 WatchSource:0}: Error finding container ca82481e2b0c7a085a715e23580a7ec47f1c53a3dee3850ac881cf0e1c95eb66: Status 404 returned error can't find the container with id ca82481e2b0c7a085a715e23580a7ec47f1c53a3dee3850ac881cf0e1c95eb66
Nov 03 00:29:22.980055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.980030    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"]
Nov 03 00:29:22.985754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:22.985735    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.001171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.001144    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-2af7b0-openshift-default-dockercfg-hwr76\""
Nov 03 00:29:23.020210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.019332    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"]
Nov 03 00:29:23.064398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064353    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-certs\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064415    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-podinfo\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064502    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-socket\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064556    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-token\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064689    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-data\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064747    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-envoy\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064802    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-credential-socket\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064857    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cczlz\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-kube-api-access-cczlz\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.064909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.064888    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istiod-ca-cert\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.166553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166519    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-credential-socket\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.166737 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166565    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cczlz\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-kube-api-access-cczlz\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.166737 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166590    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istiod-ca-cert\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.166737 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166720    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-certs\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.166914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166755    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-podinfo\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.166914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166789    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-socket\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.166914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166844    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-token\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.167080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.166979    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-data\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.167080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.167001    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-credential-socket\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.167192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.167132    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-certs\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.167192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.167163    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-socket\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.167319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.167201    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-envoy\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.167356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.167340    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-data\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.167467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.167449    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istiod-ca-cert\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.169394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.169376    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-envoy\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.169550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.169511    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-podinfo\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.185875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.185848    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cczlz\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-kube-api-access-cczlz\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.185875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.185861    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-token\") pod \"e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") " pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.308372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.306554    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:23.643289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.643218    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252" event={"ID":"67b3b3d1-a353-4a58-aadb-0e845ed8e755","Type":"ContainerStarted","Data":"3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9"}
Nov 03 00:29:23.643471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.643329    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252" event={"ID":"67b3b3d1-a353-4a58-aadb-0e845ed8e755","Type":"ContainerStarted","Data":"ca82481e2b0c7a085a715e23580a7ec47f1c53a3dee3850ac881cf0e1c95eb66"}
Nov 03 00:29:23.643471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.643362    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:23.843163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.843116    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252" podStartSLOduration=1.3620449780000001 podStartE2EDuration="1.843101069s" totalImagesPullingTime="481.056091ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:23.778640505 +0000 UTC m=+1401.333992532" watchObservedRunningTime="2026-11-03 00:29:23.843101069 +0000 UTC m=+1401.398453140"
Nov 03 00:29:23.844908 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:23.844858    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e160351_b07e_4e4d_a7ae_e7096a3912a3.slice/crio-0f04638cf68c1adf87bc564855dad7af30c19809a575ecd3e104b16f9cc57a87 WatchSource:0}: Error finding container 0f04638cf68c1adf87bc564855dad7af30c19809a575ecd3e104b16f9cc57a87: Status 404 returned error can't find the container with id 0f04638cf68c1adf87bc564855dad7af30c19809a575ecd3e104b16f9cc57a87
Nov 03 00:29:23.845448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.845425    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"]
Nov 03 00:29:23.847837 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.847809    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:23.847933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.847867    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:23.847933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:23.847905    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:24.691738 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:24.691698    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" event={"ID":"9e160351-b07e-4e4d-a7ae-e7096a3912a3","Type":"ContainerStarted","Data":"a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746"}
Nov 03 00:29:24.692277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:24.691769    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" event={"ID":"9e160351-b07e-4e4d-a7ae-e7096a3912a3","Type":"ContainerStarted","Data":"0f04638cf68c1adf87bc564855dad7af30c19809a575ecd3e104b16f9cc57a87"}
Nov 03 00:29:24.734494 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:24.734444    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podStartSLOduration=2.734429362 podStartE2EDuration="2.734429362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:24.730499629 +0000 UTC m=+1402.285851652" watchObservedRunningTime="2026-11-03 00:29:24.734429362 +0000 UTC m=+1402.289781385"
Nov 03 00:29:25.307992 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:25.307960    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:25.313484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:25.313457    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:25.698871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:25.698841    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:25.700012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:25.699986    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:29:25.756102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:25.756063    6000 patch_prober.go:29] interesting pod/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:29:25.756320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:25.756128    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:29:27.933919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:27.933857    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:29:28.003999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.003950    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-token\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004030    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-kube-api-access-kfntz\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004090    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-socket\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004177    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-data\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004298    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-envoy\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004348    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-credential-socket\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004390    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istiod-ca-cert\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004448    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-certs\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004515    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-data" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:28.004607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004532    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-podinfo\") pod \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\" (UID: \"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb\") "
Nov 03 00:29:28.004761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004696    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-credential-socket" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:28.004867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004840    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-certs" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:28.004934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004911    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istiod-ca-cert" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:28.005006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.004971    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-socket" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:28.005327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.005304    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.005451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.005330    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.005451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.005343    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.005451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.005359    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.005451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.005372    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.019466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.019405    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-podinfo" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:29:28.019730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.019706    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-kube-api-access-kfntz" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "kube-api-access-kfntz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:28.022689 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.020690    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-envoy" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:29:28.022689 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.021033    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-token" pod "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" (UID: "8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:28.107381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.107338    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.107381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.107380    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.107589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.107395    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfntz\" (UniqueName: \"kubernetes.io/projected/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-kube-api-access-kfntz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.107589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.107410    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:28.719055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.719019    6000 generic.go:356] "Generic (PLEG): container finished" podID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerID="546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5" exitCode=0
Nov 03 00:29:28.719235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.719076    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"
Nov 03 00:29:28.719235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.719106    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" event={"ID":"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb","Type":"ContainerDied","Data":"546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5"}
Nov 03 00:29:28.719235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.719165    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf" event={"ID":"8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb","Type":"ContainerDied","Data":"2ed1a73e022b74f0426583f5f455be85e6d5d04b557a7b2903e19635280494e6"}
Nov 03 00:29:28.719235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.719201    6000 scope.go:122] "RemoveContainer" containerID="546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5"
Nov 03 00:29:28.729874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.729857    6000 scope.go:122] "RemoveContainer" containerID="86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3"
Nov 03 00:29:28.740532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.740509    6000 scope.go:122] "RemoveContainer" containerID="546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5"
Nov 03 00:29:28.740846 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:29:28.740821    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5\": container with ID starting with 546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5 not found: ID does not exist" containerID="546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5"
Nov 03 00:29:28.740932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.740858    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5"} err="failed to get container status \"546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5\": rpc error: code = NotFound desc = could not find container \"546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5\": container with ID starting with 546766ec1f95ee40b88713ce2dd13ad563e7d23823b94560767ccb8d4a1e59a5 not found: ID does not exist"
Nov 03 00:29:28.740932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.740876    6000 scope.go:122] "RemoveContainer" containerID="86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3"
Nov 03 00:29:28.741168 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:29:28.741146    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3\": container with ID starting with 86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3 not found: ID does not exist" containerID="86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3"
Nov 03 00:29:28.741228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.741178    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3"} err="failed to get container status \"86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3\": rpc error: code = NotFound desc = could not find container \"86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3\": container with ID starting with 86df64c86329a639da55c91dc46fb74f732b9fb1a418b76bd958330fedd536e3 not found: ID does not exist"
Nov 03 00:29:28.783158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.783125    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"]
Nov 03 00:29:28.794477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.794449    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-3437aa-openshift-default-67b59dfbf-cswvf"]
Nov 03 00:29:28.952167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.952137    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"]
Nov 03 00:29:28.952898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.952872    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy"
Nov 03 00:29:28.952898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.952892    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy"
Nov 03 00:29:28.952995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.952921    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy"
Nov 03 00:29:28.952995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.952926    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy"
Nov 03 00:29:28.953071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.953046    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy"
Nov 03 00:29:28.958037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.958019    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:28.974480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.974460    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"]
Nov 03 00:29:28.975699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:28.975679    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-2af7b0\""
Nov 03 00:29:29.005516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.005481    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"]
Nov 03 00:29:29.006575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.006548    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" containerName="istio-proxy"
Nov 03 00:29:29.010519 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.010474    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.021157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.021132    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n4cl\" (UniqueName: \"kubernetes.io/projected/368ef1e3-6388-4152-8c7d-67cdcc5f6914-kube-api-access-2n4cl\") pod \"payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:29.021403 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.021381    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ef1e3-6388-4152-8c7d-67cdcc5f6914-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:29.025945 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.025461    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-2af7b0-dockercfg-ksbt7\""
Nov 03 00:29:29.028048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.028026    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"]
Nov 03 00:29:29.134073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.124111    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.134073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.124209    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfwpw\" (UniqueName: \"kubernetes.io/projected/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-kube-api-access-dfwpw\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.134073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.124342    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ef1e3-6388-4152-8c7d-67cdcc5f6914-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:29.134073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.124536    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n4cl\" (UniqueName: \"kubernetes.io/projected/368ef1e3-6388-4152-8c7d-67cdcc5f6914-kube-api-access-2n4cl\") pod \"payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:29.270378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.241034    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.270378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.241122    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfwpw\" (UniqueName: \"kubernetes.io/projected/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-kube-api-access-dfwpw\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.270378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.242616    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.270378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.243048    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ef1e3-6388-4152-8c7d-67cdcc5f6914-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:29.270378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.244183    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n4cl\" (UniqueName: \"kubernetes.io/projected/368ef1e3-6388-4152-8c7d-67cdcc5f6914-kube-api-access-2n4cl\") pod \"payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:29.270378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.268805    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfwpw\" (UniqueName: \"kubernetes.io/projected/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-kube-api-access-dfwpw\") pod \"payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.285670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.273802    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:29.336212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.329650    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:29.658811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.658786    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"]
Nov 03 00:29:29.662444 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:29.662409    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368ef1e3_6388_4152_8c7d_67cdcc5f6914.slice/crio-df04b0ca1e7bb7260f5976e9edaa5059e7c6525eae46cd54f3fc4bb196887689 WatchSource:0}: Error finding container df04b0ca1e7bb7260f5976e9edaa5059e7c6525eae46cd54f3fc4bb196887689: Status 404 returned error can't find the container with id df04b0ca1e7bb7260f5976e9edaa5059e7c6525eae46cd54f3fc4bb196887689
Nov 03 00:29:29.688352 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:29.688318    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f55bd92_2ed6_48a6_bca1_d5dca7ec4a3a.slice/crio-b1f9e5e35f93f9b004ac81751d6f97a31e579bde7e0166931eeb1ebbaf6b61b9 WatchSource:0}: Error finding container b1f9e5e35f93f9b004ac81751d6f97a31e579bde7e0166931eeb1ebbaf6b61b9: Status 404 returned error can't find the container with id b1f9e5e35f93f9b004ac81751d6f97a31e579bde7e0166931eeb1ebbaf6b61b9
Nov 03 00:29:29.688795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.688769    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"]
Nov 03 00:29:29.697627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.697603    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:29:29.732357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.732318    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d" event={"ID":"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a","Type":"ContainerStarted","Data":"b1f9e5e35f93f9b004ac81751d6f97a31e579bde7e0166931eeb1ebbaf6b61b9"}
Nov 03 00:29:29.741409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:29.741381    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5" event={"ID":"368ef1e3-6388-4152-8c7d-67cdcc5f6914","Type":"ContainerStarted","Data":"df04b0ca1e7bb7260f5976e9edaa5059e7c6525eae46cd54f3fc4bb196887689"}
Nov 03 00:29:30.568521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.568488    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb" path="/var/lib/kubelet/pods/8dcdb8b2-5663-49e7-b9d7-ab7eac7929cb/volumes"
Nov 03 00:29:30.674286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.674236    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:29:30.749827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.749789    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d" event={"ID":"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a","Type":"ContainerStarted","Data":"9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402"}
Nov 03 00:29:30.750004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.749856    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:30.751483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.751453    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5" event={"ID":"368ef1e3-6388-4152-8c7d-67cdcc5f6914","Type":"ContainerStarted","Data":"0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9"}
Nov 03 00:29:30.751601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.751499    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:30.790449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.790393    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d" podStartSLOduration=2.790376719 podStartE2EDuration="2.790376719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:30.785558241 +0000 UTC m=+1408.340910265" watchObservedRunningTime="2026-11-03 00:29:30.790376719 +0000 UTC m=+1408.345728744"
Nov 03 00:29:30.822059 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:30.821880    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5" podStartSLOduration=2.821862254 podStartE2EDuration="2.821862254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:30.821166498 +0000 UTC m=+1408.376518522" watchObservedRunningTime="2026-11-03 00:29:30.821862254 +0000 UTC m=+1408.377214279"
Nov 03 00:29:31.608966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:31.608930    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"]
Nov 03 00:29:36.091817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.091788    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"]
Nov 03 00:29:36.092241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.091974    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4" podUID="05c48858-5f75-4623-87d7-557a220968a1" containerName="maas-api" containerID="cri-o://a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298" gracePeriod=30
Nov 03 00:29:36.140524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.140483    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4" podUID="05c48858-5f75-4623-87d7-557a220968a1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.115:8443/health\": dial tcp 10.134.0.115:8443: connect: connection refused"
Nov 03 00:29:36.395217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.395189    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:29:36.467492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.467460    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-metrics-certs\") pod \"05c48858-5f75-4623-87d7-557a220968a1\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") "
Nov 03 00:29:36.467701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.467558    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05c48858-5f75-4623-87d7-557a220968a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c48858-5f75-4623-87d7-557a220968a1-openshift-service-ca\") pod \"05c48858-5f75-4623-87d7-557a220968a1\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") "
Nov 03 00:29:36.467808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.467790    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05c48858-5f75-4623-87d7-557a220968a1-kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/05c48858-5f75-4623-87d7-557a220968a1-kube-api-access-b5f6r\") pod \"05c48858-5f75-4623-87d7-557a220968a1\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") "
Nov 03 00:29:36.467867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.467844    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-maas-api-tls\") pod \"05c48858-5f75-4623-87d7-557a220968a1\" (UID: \"05c48858-5f75-4623-87d7-557a220968a1\") "
Nov 03 00:29:36.467919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.467892    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05c48858-5f75-4623-87d7-557a220968a1-openshift-service-ca" pod "05c48858-5f75-4623-87d7-557a220968a1" (UID: "05c48858-5f75-4623-87d7-557a220968a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:36.468506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.468480    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05c48858-5f75-4623-87d7-557a220968a1-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:36.470750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.470721    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-maas-api-tls" pod "05c48858-5f75-4623-87d7-557a220968a1" (UID: "05c48858-5f75-4623-87d7-557a220968a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:29:36.470956 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.470932    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05c48858-5f75-4623-87d7-557a220968a1-kube-api-access-b5f6r" pod "05c48858-5f75-4623-87d7-557a220968a1" (UID: "05c48858-5f75-4623-87d7-557a220968a1"). InnerVolumeSpecName "kube-api-access-b5f6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:36.471442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.471419    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-metrics-certs" pod "05c48858-5f75-4623-87d7-557a220968a1" (UID: "05c48858-5f75-4623-87d7-557a220968a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:29:36.583993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.573916    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:36.583993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.583904    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5f6r\" (UniqueName: \"kubernetes.io/projected/05c48858-5f75-4623-87d7-557a220968a1-kube-api-access-b5f6r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:36.583993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.583964    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05c48858-5f75-4623-87d7-557a220968a1-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:36.648370 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.632327    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4" podUID="8cde5e87-df70-4e15-8e36-bf7587b15550" containerName="payload-pre-processing" containerID="cri-o://930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790" gracePeriod=25
Nov 03 00:29:36.813775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.813741    6000 generic.go:356] "Generic (PLEG): container finished" podID="05c48858-5f75-4623-87d7-557a220968a1" containerID="a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298" exitCode=0
Nov 03 00:29:36.813972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.813928    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4" event={"ID":"05c48858-5f75-4623-87d7-557a220968a1","Type":"ContainerDied","Data":"a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298"}
Nov 03 00:29:36.813972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.813965    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4" event={"ID":"05c48858-5f75-4623-87d7-557a220968a1","Type":"ContainerDied","Data":"84affccb18f1f5394b8abbee438fb17a61f62d29d57ab1e9fed3e164711832fe"}
Nov 03 00:29:36.814105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.813986    6000 scope.go:122] "RemoveContainer" containerID="a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298"
Nov 03 00:29:36.814225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.814209    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"
Nov 03 00:29:36.825497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.825474    6000 scope.go:122] "RemoveContainer" containerID="a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298"
Nov 03 00:29:36.825771 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:29:36.825746    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298\": container with ID starting with a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298 not found: ID does not exist" containerID="a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298"
Nov 03 00:29:36.825857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.825784    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298"} err="failed to get container status \"a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298\": rpc error: code = NotFound desc = could not find container \"a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298\": container with ID starting with a712838fe98534e727076a56210333181899c0f58be65866a2e2b7dd7bc15298 not found: ID does not exist"
Nov 03 00:29:36.868059 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.868031    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"]
Nov 03 00:29:36.871653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.871626    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e37c9d-5c78f945f8-7nmt4"]
Nov 03 00:29:36.972674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:36.972634    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:29:37.095056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.095015    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cde5e87-df70-4e15-8e36-bf7587b15550-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cde5e87-df70-4e15-8e36-bf7587b15550-plugins-config-volume\") pod \"8cde5e87-df70-4e15-8e36-bf7587b15550\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") "
Nov 03 00:29:37.096237 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.096151    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cde5e87-df70-4e15-8e36-bf7587b15550-plugins-config-volume" pod "8cde5e87-df70-4e15-8e36-bf7587b15550" (UID: "8cde5e87-df70-4e15-8e36-bf7587b15550"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:29:37.096644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.096617    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cde5e87-df70-4e15-8e36-bf7587b15550-kube-api-access-gfnkq\" (UniqueName: \"kubernetes.io/projected/8cde5e87-df70-4e15-8e36-bf7587b15550-kube-api-access-gfnkq\") pod \"8cde5e87-df70-4e15-8e36-bf7587b15550\" (UID: \"8cde5e87-df70-4e15-8e36-bf7587b15550\") "
Nov 03 00:29:37.097684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.097320    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cde5e87-df70-4e15-8e36-bf7587b15550-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:37.131126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.131033    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cde5e87-df70-4e15-8e36-bf7587b15550-kube-api-access-gfnkq" pod "8cde5e87-df70-4e15-8e36-bf7587b15550" (UID: "8cde5e87-df70-4e15-8e36-bf7587b15550"). InnerVolumeSpecName "kube-api-access-gfnkq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:29:37.198621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.198584    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gfnkq\" (UniqueName: \"kubernetes.io/projected/8cde5e87-df70-4e15-8e36-bf7587b15550-kube-api-access-gfnkq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:29:37.822363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.822325    6000 generic.go:356] "Generic (PLEG): container finished" podID="8cde5e87-df70-4e15-8e36-bf7587b15550" containerID="930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790" exitCode=0
Nov 03 00:29:37.822555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.822373    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"
Nov 03 00:29:37.822555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.822406    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4" event={"ID":"8cde5e87-df70-4e15-8e36-bf7587b15550","Type":"ContainerDied","Data":"930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790"}
Nov 03 00:29:37.822555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.822456    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4" event={"ID":"8cde5e87-df70-4e15-8e36-bf7587b15550","Type":"ContainerDied","Data":"163ba2d650c7c81bfd8c49b782abf309158b79535038bbc72ea87ef4adecf8f1"}
Nov 03 00:29:37.822555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.822479    6000 scope.go:122] "RemoveContainer" containerID="930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790"
Nov 03 00:29:37.833133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.833117    6000 scope.go:122] "RemoveContainer" containerID="930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790"
Nov 03 00:29:37.833436 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:29:37.833415    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790\": container with ID starting with 930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790 not found: ID does not exist" containerID="930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790"
Nov 03 00:29:37.833436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.833442    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790"} err="failed to get container status \"930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790\": rpc error: code = NotFound desc = could not find container \"930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790\": container with ID starting with 930602684ebbf031394921d64b73e863241b108c22492369683396df9a3fe790 not found: ID does not exist"
Nov 03 00:29:37.871793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.871763    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"]
Nov 03 00:29:37.883211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:37.883189    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-846492-c67886c9f-fwcz4"]
Nov 03 00:29:38.568414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:38.568382    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05c48858-5f75-4623-87d7-557a220968a1" path="/var/lib/kubelet/pods/05c48858-5f75-4623-87d7-557a220968a1/volumes"
Nov 03 00:29:38.568867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:38.568852    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cde5e87-df70-4e15-8e36-bf7587b15550" path="/var/lib/kubelet/pods/8cde5e87-df70-4e15-8e36-bf7587b15550/volumes"
Nov 03 00:29:41.765686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:41.765653    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:29:41.766139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:41.765751    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:29:46.476124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.476088    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"]
Nov 03 00:29:46.476792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.476775    6000 cpu_manager.go:414] "containerMap: removing container" podUID="05c48858-5f75-4623-87d7-557a220968a1" containerName="maas-api"
Nov 03 00:29:46.476882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.476793    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05c48858-5f75-4623-87d7-557a220968a1" containerName="maas-api"
Nov 03 00:29:46.476882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.476822    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8cde5e87-df70-4e15-8e36-bf7587b15550" containerName="payload-pre-processing"
Nov 03 00:29:46.476882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.476828    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cde5e87-df70-4e15-8e36-bf7587b15550" containerName="payload-pre-processing"
Nov 03 00:29:46.476982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.476912    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="05c48858-5f75-4623-87d7-557a220968a1" containerName="maas-api"
Nov 03 00:29:46.476982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.476926    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cde5e87-df70-4e15-8e36-bf7587b15550" containerName="payload-pre-processing"
Nov 03 00:29:46.481683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.481663    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:46.500646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.500619    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"]
Nov 03 00:29:46.501683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.501651    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ea275f\""
Nov 03 00:29:46.642282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.641660    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7aba0-4c8d-4116-a042-1583b36a49f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:46.642282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.641995    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pfcp\" (UniqueName: \"kubernetes.io/projected/c4b7aba0-4c8d-4116-a042-1583b36a49f4-kube-api-access-6pfcp\") pod \"payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:46.830335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.754782    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pfcp\" (UniqueName: \"kubernetes.io/projected/c4b7aba0-4c8d-4116-a042-1583b36a49f4-kube-api-access-6pfcp\") pod \"payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:46.830718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.830688    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7aba0-4c8d-4116-a042-1583b36a49f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:46.856346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.840755    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7aba0-4c8d-4116-a042-1583b36a49f4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:46.956572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:46.956501    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pfcp\" (UniqueName: \"kubernetes.io/projected/c4b7aba0-4c8d-4116-a042-1583b36a49f4-kube-api-access-6pfcp\") pod \"payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:47.107331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:47.102756    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:47.578181 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:47.578145    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b7aba0_4c8d_4116_a042_1583b36a49f4.slice/crio-5b90fa72643f04b15ab8d6fce8c2b17f009fb7139fb8ebafacda090f97d543cc WatchSource:0}: Error finding container 5b90fa72643f04b15ab8d6fce8c2b17f009fb7139fb8ebafacda090f97d543cc: Status 404 returned error can't find the container with id 5b90fa72643f04b15ab8d6fce8c2b17f009fb7139fb8ebafacda090f97d543cc
Nov 03 00:29:47.578667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:47.578328    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"]
Nov 03 00:29:48.109371 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.109332    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"]
Nov 03 00:29:48.113888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.113862    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.119732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.119707    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk" event={"ID":"c4b7aba0-4c8d-4116-a042-1583b36a49f4","Type":"ContainerStarted","Data":"6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982"}
Nov 03 00:29:48.119872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.119757    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk" event={"ID":"c4b7aba0-4c8d-4116-a042-1583b36a49f4","Type":"ContainerStarted","Data":"5b90fa72643f04b15ab8d6fce8c2b17f009fb7139fb8ebafacda090f97d543cc"}
Nov 03 00:29:48.119872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.119789    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:29:48.134942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.134915    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"]
Nov 03 00:29:48.147897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.147873    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-2af7b0-serving-cert\""
Nov 03 00:29:48.210318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.210243    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk" podStartSLOduration=2.210224446 podStartE2EDuration="2.210224446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:48.206830896 +0000 UTC m=+1425.762182919" watchObservedRunningTime="2026-11-03 00:29:48.210224446 +0000 UTC m=+1425.765576488"
Nov 03 00:29:48.222717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.222682    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zpk8l\" (UniqueName: \"kubernetes.io/projected/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-kube-api-access-zpk8l\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.222862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.222837    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.223009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.222990    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-metrics-certs\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.223192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.223167    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.324480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.324441    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-metrics-certs\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.324668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.324558    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.324668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.324656    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zpk8l\" (UniqueName: \"kubernetes.io/projected/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-kube-api-access-zpk8l\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.324824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.324726    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.325357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.325241    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.327318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.327294    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-maas-api-tls\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.327406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.327294    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-metrics-certs\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.345969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.345941    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zpk8l\" (UniqueName: \"kubernetes.io/projected/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-kube-api-access-zpk8l\") pod \"maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.438082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.436550    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:48.812754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:48.812724    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"]
Nov 03 00:29:48.813130 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:48.813012    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e879dc1_80a5_47c3_a3e1_8108b64dc9ca.slice/crio-81d56e7dfaa30809a9326877088c4f808c02e4189d3752ce863e9d03405cd6ab WatchSource:0}: Error finding container 81d56e7dfaa30809a9326877088c4f808c02e4189d3752ce863e9d03405cd6ab: Status 404 returned error can't find the container with id 81d56e7dfaa30809a9326877088c4f808c02e4189d3752ce863e9d03405cd6ab
Nov 03 00:29:49.128767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.128731    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg" event={"ID":"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca","Type":"ContainerStarted","Data":"81d56e7dfaa30809a9326877088c4f808c02e4189d3752ce863e9d03405cd6ab"}
Nov 03 00:29:49.658827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.658794    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"]
Nov 03 00:29:49.665578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.665553    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.694084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.688724    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"]
Nov 03 00:29:49.694084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.693015    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-2df502-openshift-default-dockercfg-rlpl8\""
Nov 03 00:29:49.735621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.735585    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f097aed8-d75f-4dda-ac53-27eb137685a5-istiod-ca-cert\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.735909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.735637    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-token\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.735909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.735836    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-data\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.736010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.735982    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-credential-socket\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.736099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.736079    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-podinfo\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.736169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.736131    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5z4dv\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-kube-api-access-5z4dv\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.736303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.736247    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-envoy\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.736363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.736317    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-socket\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.736433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.736409    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-certs\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.838864    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-envoy\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.838940    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-socket\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.839021    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-certs\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.839186    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f097aed8-d75f-4dda-ac53-27eb137685a5-istiod-ca-cert\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.839222    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-token\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.839377    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-data\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.839436    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-credential-socket\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.839473    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-podinfo\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.844056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.839514    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5z4dv\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-kube-api-access-5z4dv\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.868300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.860652    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-socket\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.868300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.860955    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-certs\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.868300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.864616    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-data\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.868300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.864879    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-credential-socket\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.895290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.868850    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f097aed8-d75f-4dda-ac53-27eb137685a5-istiod-ca-cert\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.895290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.875535    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-envoy\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:49.944879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:49.944777    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5z4dv\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-kube-api-access-5z4dv\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:50.013884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.013816    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-podinfo\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:50.074957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.045880    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-token\") pod \"e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") " pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:50.155280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.149575    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg" event={"ID":"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca","Type":"ContainerStarted","Data":"a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715"}
Nov 03 00:29:50.155280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.151179    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:50.171907 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:29:50.171860    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:29:50.196137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.196036    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg" podStartSLOduration=1.860319622 podStartE2EDuration="2.196020649s" totalImagesPullingTime="335.701027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:50.193863205 +0000 UTC m=+1427.749215230" watchObservedRunningTime="2026-11-03 00:29:50.196020649 +0000 UTC m=+1427.751372674"
Nov 03 00:29:50.287136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.287105    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:50.682644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.682613    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"]
Nov 03 00:29:50.683097 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:50.683068    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf097aed8_d75f_4dda_ac53_27eb137685a5.slice/crio-5f99a54adffddabbfb9d5c6b227480e05909c61979a2da766f81556931abc45c WatchSource:0}: Error finding container 5f99a54adffddabbfb9d5c6b227480e05909c61979a2da766f81556931abc45c: Status 404 returned error can't find the container with id 5f99a54adffddabbfb9d5c6b227480e05909c61979a2da766f81556931abc45c
Nov 03 00:29:50.686045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.686007    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:50.686110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.686091    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:50.686151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:50.686136    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:29:51.159922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:51.159889    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" event={"ID":"f097aed8-d75f-4dda-ac53-27eb137685a5","Type":"ContainerStarted","Data":"1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae"}
Nov 03 00:29:51.160477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:51.159942    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" event={"ID":"f097aed8-d75f-4dda-ac53-27eb137685a5","Type":"ContainerStarted","Data":"5f99a54adffddabbfb9d5c6b227480e05909c61979a2da766f81556931abc45c"}
Nov 03 00:29:51.224206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:51.224157    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podStartSLOduration=2.224146249 podStartE2EDuration="2.224146249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:51.211923687 +0000 UTC m=+1428.767275733" watchObservedRunningTime="2026-11-03 00:29:51.224146249 +0000 UTC m=+1428.779498272"
Nov 03 00:29:51.287958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:51.287917    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:51.293878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:51.293852    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:51.810281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:51.806798    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:29:52.358175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:52.358145    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:52.392341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:52.392311    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:29:53.010357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.010319    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"]
Nov 03 00:29:53.015077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.015056    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.032350 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.032320    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"]
Nov 03 00:29:53.071291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.068679    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-191362\""
Nov 03 00:29:53.161287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.160778    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-191362-dockercfg-6p7pn\""
Nov 03 00:29:53.233278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.232230    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-575k4\" (UniqueName: \"kubernetes.io/projected/d71acda9-cc37-46c4-8363-6247e99cd612-kube-api-access-575k4\") pod \"payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.233278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.232409    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71acda9-cc37-46c4-8363-6247e99cd612-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.340581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.338177    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-575k4\" (UniqueName: \"kubernetes.io/projected/d71acda9-cc37-46c4-8363-6247e99cd612-kube-api-access-575k4\") pod \"payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.340581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.338360    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71acda9-cc37-46c4-8363-6247e99cd612-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.344858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.344821    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71acda9-cc37-46c4-8363-6247e99cd612-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.362487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.362452    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-575k4\" (UniqueName: \"kubernetes.io/projected/d71acda9-cc37-46c4-8363-6247e99cd612-kube-api-access-575k4\") pod \"payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.629499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.629467    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:53.938196 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:29:53.938160    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71acda9_cc37_46c4_8363_6247e99cd612.slice/crio-7835ec379ef82223ce6224a3eb02baff9deb0ed07d7c611fd3a60127c0fbcdb1 WatchSource:0}: Error finding container 7835ec379ef82223ce6224a3eb02baff9deb0ed07d7c611fd3a60127c0fbcdb1: Status 404 returned error can't find the container with id 7835ec379ef82223ce6224a3eb02baff9deb0ed07d7c611fd3a60127c0fbcdb1
Nov 03 00:29:53.938376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:53.938357    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"]
Nov 03 00:29:54.369038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:54.369003    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x" event={"ID":"d71acda9-cc37-46c4-8363-6247e99cd612","Type":"ContainerStarted","Data":"f9ec3259441092d297dea7060c9211a5dc0c6efbae1ffbc98555550cf09125ee"}
Nov 03 00:29:54.369596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:54.369069    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:29:54.369596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:54.369089    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x" event={"ID":"d71acda9-cc37-46c4-8363-6247e99cd612","Type":"ContainerStarted","Data":"7835ec379ef82223ce6224a3eb02baff9deb0ed07d7c611fd3a60127c0fbcdb1"}
Nov 03 00:29:54.409269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:54.409213    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x" podStartSLOduration=2.409199367 podStartE2EDuration="2.409199367s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:29:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:29:54.407132664 +0000 UTC m=+1431.962484688" watchObservedRunningTime="2026-11-03 00:29:54.409199367 +0000 UTC m=+1431.964551390"
Nov 03 00:29:57.364650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:57.364623    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:29:59.131853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:29:59.131820    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:30:05.380390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:05.380353    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:30:10.138834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.138794    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"]
Nov 03 00:30:10.145221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.145202    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.159434 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.159405    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-b6cd31\""
Nov 03 00:30:10.162987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.162961    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"]
Nov 03 00:30:10.193959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.193914    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"]
Nov 03 00:30:10.199792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.199680    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:10.216153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.216118    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"]
Nov 03 00:30:10.260626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.260586    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0964cb-09f8-4208-a29d-0162bc764ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.277288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.261761    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-b6cd31-dockercfg-94dhs\""
Nov 03 00:30:10.279382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.279338    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq8fw\" (UniqueName: \"kubernetes.io/projected/3f0964cb-09f8-4208-a29d-0162bc764ac1-kube-api-access-bq8fw\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.421369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.414953    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0964cb-09f8-4208-a29d-0162bc764ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.430276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.421648    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2893fb17-1482-4e46-ae4d-01355e048194-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:10.430580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.430547    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq8fw\" (UniqueName: \"kubernetes.io/projected/3f0964cb-09f8-4208-a29d-0162bc764ac1-kube-api-access-bq8fw\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.446322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.434367    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86hrk\" (UniqueName: \"kubernetes.io/projected/2893fb17-1482-4e46-ae4d-01355e048194-kube-api-access-86hrk\") pod \"payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:10.447197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.444799    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0964cb-09f8-4208-a29d-0162bc764ac1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.476477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.476444    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq8fw\" (UniqueName: \"kubernetes.io/projected/3f0964cb-09f8-4208-a29d-0162bc764ac1-kube-api-access-bq8fw\") pod \"payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.548500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.548471    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2893fb17-1482-4e46-ae4d-01355e048194-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:10.548688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.548517    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86hrk\" (UniqueName: \"kubernetes.io/projected/2893fb17-1482-4e46-ae4d-01355e048194-kube-api-access-86hrk\") pod \"payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:10.549218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.549182    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2893fb17-1482-4e46-ae4d-01355e048194-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:10.567838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.567809    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86hrk\" (UniqueName: \"kubernetes.io/projected/2893fb17-1482-4e46-ae4d-01355e048194-kube-api-access-86hrk\") pod \"payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:10.759583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.759497    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:10.861808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:10.861756    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:11.063513 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:30:11.063472    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f0964cb_09f8_4208_a29d_0162bc764ac1.slice/crio-60546b738d016dd959dd3abca451c5330b41211cf50ef05aecee7791960a235d WatchSource:0}: Error finding container 60546b738d016dd959dd3abca451c5330b41211cf50ef05aecee7791960a235d: Status 404 returned error can't find the container with id 60546b738d016dd959dd3abca451c5330b41211cf50ef05aecee7791960a235d
Nov 03 00:30:11.064786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.064675    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"]
Nov 03 00:30:11.218228 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:30:11.218204    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2893fb17_1482_4e46_ae4d_01355e048194.slice/crio-4c1175b7695e4e5ae8839b166d1e83030a5cf6ec8500b7efb640754c8fdcb9ca WatchSource:0}: Error finding container 4c1175b7695e4e5ae8839b166d1e83030a5cf6ec8500b7efb640754c8fdcb9ca: Status 404 returned error can't find the container with id 4c1175b7695e4e5ae8839b166d1e83030a5cf6ec8500b7efb640754c8fdcb9ca
Nov 03 00:30:11.218652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.218490    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"]
Nov 03 00:30:11.561326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.561286    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r" event={"ID":"3f0964cb-09f8-4208-a29d-0162bc764ac1","Type":"ContainerStarted","Data":"1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d"}
Nov 03 00:30:11.561526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.561354    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r" event={"ID":"3f0964cb-09f8-4208-a29d-0162bc764ac1","Type":"ContainerStarted","Data":"60546b738d016dd959dd3abca451c5330b41211cf50ef05aecee7791960a235d"}
Nov 03 00:30:11.561526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.561413    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:11.563132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.563105    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt" event={"ID":"2893fb17-1482-4e46-ae4d-01355e048194","Type":"ContainerStarted","Data":"d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba"}
Nov 03 00:30:11.563311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.563146    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt" event={"ID":"2893fb17-1482-4e46-ae4d-01355e048194","Type":"ContainerStarted","Data":"4c1175b7695e4e5ae8839b166d1e83030a5cf6ec8500b7efb640754c8fdcb9ca"}
Nov 03 00:30:11.563311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.563197    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:11.600683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.600623    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r" podStartSLOduration=1.60060462 podStartE2EDuration="1.60060462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:11.597761376 +0000 UTC m=+1449.153113402" watchObservedRunningTime="2026-11-03 00:30:11.60060462 +0000 UTC m=+1449.155956650"
Nov 03 00:30:11.638034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:11.637990    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt" podStartSLOduration=1.637976514 podStartE2EDuration="1.637976514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:11.634662409 +0000 UTC m=+1449.190014449" watchObservedRunningTime="2026-11-03 00:30:11.637976514 +0000 UTC m=+1449.193328562"
Nov 03 00:30:18.118724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:18.118631    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"]
Nov 03 00:30:18.126740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:18.126706    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"]
Nov 03 00:30:18.889281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:18.888334    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"]
Nov 03 00:30:18.965091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:18.964495    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:18.985279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:18.981727    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"]
Nov 03 00:30:18.985279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:18.983093    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-2df502-serving-cert\""
Nov 03 00:30:19.146356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.123299    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.146356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.123404    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de43d331-7219-4337-a7b4-dcb90588e10f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.146356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.123570    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-metrics-certs\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.146356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.123649    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2h72\" (UniqueName: \"kubernetes.io/projected/de43d331-7219-4337-a7b4-dcb90588e10f-kube-api-access-k2h72\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.240279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.240188    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-metrics-certs\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.240488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.240332    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2h72\" (UniqueName: \"kubernetes.io/projected/de43d331-7219-4337-a7b4-dcb90588e10f-kube-api-access-k2h72\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.240557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.240539    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.242287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.240618    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de43d331-7219-4337-a7b4-dcb90588e10f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.242287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.241662    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de43d331-7219-4337-a7b4-dcb90588e10f-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.248961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.248936    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-maas-api-tls\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.249088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.248936    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-metrics-certs\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.266414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.265846    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2h72\" (UniqueName: \"kubernetes.io/projected/de43d331-7219-4337-a7b4-dcb90588e10f-kube-api-access-k2h72\") pod \"maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.510464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.509635    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:19.944180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:19.938600    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:30:20.020851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:20.014657    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"]
Nov 03 00:30:20.080181 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:30:20.080122    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:30:20.749062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:20.748765    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw" event={"ID":"de43d331-7219-4337-a7b4-dcb90588e10f","Type":"ContainerStarted","Data":"e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0"}
Nov 03 00:30:20.749062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:20.748851    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw" event={"ID":"de43d331-7219-4337-a7b4-dcb90588e10f","Type":"ContainerStarted","Data":"2ccebfddf1222adf7fb0914b4dfd56aa0295dcb33d86986d1d59a81a6c99ccea"}
Nov 03 00:30:20.758283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:20.750216    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:22.572470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:22.572445    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:30:22.572888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:22.572509    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:30:22.612320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:22.612247    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw" podStartSLOduration=4.05049481 podStartE2EDuration="4.612231821s" totalImagesPullingTime="561.737011ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:20.8181624 +0000 UTC m=+1458.373514427" watchObservedRunningTime="2026-11-03 00:30:22.612231821 +0000 UTC m=+1460.167583855"
Nov 03 00:30:23.119686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.119626    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5" podUID="368ef1e3-6388-4152-8c7d-67cdcc5f6914" containerName="payload-processing" containerID="cri-o://0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9" gracePeriod=25
Nov 03 00:30:23.127345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.127304    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d" podUID="2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" containerName="payload-pre-processing" containerID="cri-o://9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402" gracePeriod=25
Nov 03 00:30:23.439206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.439171    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:30:23.469323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.469297    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:30:23.592458 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.592420    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368ef1e3-6388-4152-8c7d-67cdcc5f6914-kube-api-access-2n4cl\" (UniqueName: \"kubernetes.io/projected/368ef1e3-6388-4152-8c7d-67cdcc5f6914-kube-api-access-2n4cl\") pod \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") "
Nov 03 00:30:23.592942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.592510    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-kube-api-access-dfwpw\" (UniqueName: \"kubernetes.io/projected/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-kube-api-access-dfwpw\") pod \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") "
Nov 03 00:30:23.592942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.592710    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-plugins-config-volume\") pod \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\" (UID: \"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a\") "
Nov 03 00:30:23.592942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.592777    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/368ef1e3-6388-4152-8c7d-67cdcc5f6914-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ef1e3-6388-4152-8c7d-67cdcc5f6914-plugins-config-volume\") pod \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\" (UID: \"368ef1e3-6388-4152-8c7d-67cdcc5f6914\") "
Nov 03 00:30:23.593224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.593202    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368ef1e3-6388-4152-8c7d-67cdcc5f6914-plugins-config-volume" pod "368ef1e3-6388-4152-8c7d-67cdcc5f6914" (UID: "368ef1e3-6388-4152-8c7d-67cdcc5f6914"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:23.593377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.593350    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-plugins-config-volume" pod "2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" (UID: "2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:23.593545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.593523    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:30:23.593608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.593552    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/368ef1e3-6388-4152-8c7d-67cdcc5f6914-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:30:23.595719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.595691    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368ef1e3-6388-4152-8c7d-67cdcc5f6914-kube-api-access-2n4cl" pod "368ef1e3-6388-4152-8c7d-67cdcc5f6914" (UID: "368ef1e3-6388-4152-8c7d-67cdcc5f6914"). InnerVolumeSpecName "kube-api-access-2n4cl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:23.596549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.596519    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-kube-api-access-dfwpw" pod "2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" (UID: "2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a"). InnerVolumeSpecName "kube-api-access-dfwpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:23.695211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.695125    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n4cl\" (UniqueName: \"kubernetes.io/projected/368ef1e3-6388-4152-8c7d-67cdcc5f6914-kube-api-access-2n4cl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:30:23.695211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.695159    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfwpw\" (UniqueName: \"kubernetes.io/projected/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a-kube-api-access-dfwpw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:30:23.782782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.782748    6000 generic.go:356] "Generic (PLEG): container finished" podID="2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" containerID="9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402" exitCode=0
Nov 03 00:30:23.782981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.782837    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"
Nov 03 00:30:23.782981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.782850    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d" event={"ID":"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a","Type":"ContainerDied","Data":"9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402"}
Nov 03 00:30:23.782981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.782908    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d" event={"ID":"2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a","Type":"ContainerDied","Data":"b1f9e5e35f93f9b004ac81751d6f97a31e579bde7e0166931eeb1ebbaf6b61b9"}
Nov 03 00:30:23.782981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.782937    6000 scope.go:122] "RemoveContainer" containerID="9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402"
Nov 03 00:30:23.786440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.786410    6000 generic.go:356] "Generic (PLEG): container finished" podID="368ef1e3-6388-4152-8c7d-67cdcc5f6914" containerID="0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9" exitCode=0
Nov 03 00:30:23.786593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.786492    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5" event={"ID":"368ef1e3-6388-4152-8c7d-67cdcc5f6914","Type":"ContainerDied","Data":"0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9"}
Nov 03 00:30:23.786593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.786539    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5" event={"ID":"368ef1e3-6388-4152-8c7d-67cdcc5f6914","Type":"ContainerDied","Data":"df04b0ca1e7bb7260f5976e9edaa5059e7c6525eae46cd54f3fc4bb196887689"}
Nov 03 00:30:23.786813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.786789    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"
Nov 03 00:30:23.795406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.795382    6000 scope.go:122] "RemoveContainer" containerID="9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402"
Nov 03 00:30:23.795752 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:30:23.795722    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402\": container with ID starting with 9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402 not found: ID does not exist" containerID="9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402"
Nov 03 00:30:23.795870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.795762    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402"} err="failed to get container status \"9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402\": rpc error: code = NotFound desc = could not find container \"9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402\": container with ID starting with 9dc7cc0b911eb844ac47e541e927e9be26adf64a5354595cfa3836f16a90a402 not found: ID does not exist"
Nov 03 00:30:23.795870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.795786    6000 scope.go:122] "RemoveContainer" containerID="0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9"
Nov 03 00:30:23.807827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.807795    6000 scope.go:122] "RemoveContainer" containerID="0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9"
Nov 03 00:30:23.808154 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:30:23.808133    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9\": container with ID starting with 0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9 not found: ID does not exist" containerID="0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9"
Nov 03 00:30:23.808214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.808163    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9"} err="failed to get container status \"0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9\": rpc error: code = NotFound desc = could not find container \"0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9\": container with ID starting with 0eb8c52813043ee492033a854bc27868ca501768e23939216935d904d0c978e9 not found: ID does not exist"
Nov 03 00:30:23.846963 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.846926    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"]
Nov 03 00:30:23.858412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.858382    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-2af7b0-7b4b9c676c-9smm5"]
Nov 03 00:30:23.917301 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.916486    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"]
Nov 03 00:30:23.943279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:23.939477    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-2af7b0-7cd577cf67-xx67d"]
Nov 03 00:30:24.569531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:24.569451    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" path="/var/lib/kubelet/pods/2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a/volumes"
Nov 03 00:30:24.569945 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:24.569932    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368ef1e3-6388-4152-8c7d-67cdcc5f6914" path="/var/lib/kubelet/pods/368ef1e3-6388-4152-8c7d-67cdcc5f6914/volumes"
Nov 03 00:30:25.701044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.701012    6000 patch_prober.go:29] interesting pod/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 00:30:25.701524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.701065    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 00:30:25.805335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.805243    6000 generic.go:356] "Generic (PLEG): container finished" podID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerID="a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746" exitCode=0
Nov 03 00:30:25.805335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.805316    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" event={"ID":"9e160351-b07e-4e4d-a7ae-e7096a3912a3","Type":"ContainerDied","Data":"a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746"}
Nov 03 00:30:25.805857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.805836    6000 scope.go:122] "RemoveContainer" containerID="a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746"
Nov 03 00:30:25.807772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.807740    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:30:25.807869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.807812    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:30:25.807869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:25.807845    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:30:26.817591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:26.817557    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" event={"ID":"9e160351-b07e-4e4d-a7ae-e7096a3912a3","Type":"ContainerStarted","Data":"9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0"}
Nov 03 00:30:27.308280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:27.307676    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:30:27.308280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:27.307817    6000 patch_prober.go:29] interesting pod/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 00:30:27.308280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:27.307863    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 00:30:27.779801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:27.779772    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:30:28.311707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:28.311664    6000 patch_prober.go:29] interesting pod/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 00:30:28.312196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:28.311748    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 00:30:29.309661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:29.309620    6000 patch_prober.go:29] interesting pod/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 00:30:29.309944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:29.309915    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 00:30:30.310641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:30.310599    6000 patch_prober.go:29] interesting pod/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 00:30:30.311318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:30.311276    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 00:30:31.311746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:31.311710    6000 patch_prober.go:29] interesting pod/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body=
Nov 03 00:30:31.315306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:31.314448    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused"
Nov 03 00:30:32.312620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:32.312587    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:30:32.313112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:32.313089    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:30:32.313885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:32.313865    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:30:36.449586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.449550    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"]
Nov 03 00:30:36.450302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.450287    6000 cpu_manager.go:414] "containerMap: removing container" podUID="368ef1e3-6388-4152-8c7d-67cdcc5f6914" containerName="payload-processing"
Nov 03 00:30:36.450360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.450303    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368ef1e3-6388-4152-8c7d-67cdcc5f6914" containerName="payload-processing"
Nov 03 00:30:36.450360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.450330    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" containerName="payload-pre-processing"
Nov 03 00:30:36.450360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.450335    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" containerName="payload-pre-processing"
Nov 03 00:30:36.450456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.450448    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="368ef1e3-6388-4152-8c7d-67cdcc5f6914" containerName="payload-processing"
Nov 03 00:30:36.450492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.450460    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f55bd92-2ed6-48a6-bca1-d5dca7ec4a3a" containerName="payload-pre-processing"
Nov 03 00:30:36.453431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.453413    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.471545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.471512    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-b6cd31-serving-cert\""
Nov 03 00:30:36.484066 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.484036    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"]
Nov 03 00:30:36.561783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.561745    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b760644-262e-4470-b3a2-e462d3479d80-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.561970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.561806    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz8kv\" (UniqueName: \"kubernetes.io/projected/2b760644-262e-4470-b3a2-e462d3479d80-kube-api-access-mz8kv\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.562032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.562002    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-metrics-certs\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.562113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.562098    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.677445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.677399    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-metrics-certs\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.685750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.685714    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.696511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.686688    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b760644-262e-4470-b3a2-e462d3479d80-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.696865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.696837    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz8kv\" (UniqueName: \"kubernetes.io/projected/2b760644-262e-4470-b3a2-e462d3479d80-kube-api-access-mz8kv\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.697772 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:30:36.686382    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-serving-cert: secret "maas-api-e2e-ipp-b-b6cd31-serving-cert" not found
Nov 03 00:30:36.697772 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:30:36.697580    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls podName:2b760644-262e-4470-b3a2-e462d3479d80 nodeName:}" failed. No retries permitted until 2026-11-03 00:30:37.197555722 +0000 UTC m=+1474.752907723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls") pod "maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz" (UID: "2b760644-262e-4470-b3a2-e462d3479d80") : secret "maas-api-e2e-ipp-b-b6cd31-serving-cert" not found
Nov 03 00:30:36.697985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.688580    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b760644-262e-4470-b3a2-e462d3479d80-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.727031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.726918    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz8kv\" (UniqueName: \"kubernetes.io/projected/2b760644-262e-4470-b3a2-e462d3479d80-kube-api-access-mz8kv\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:36.736583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:36.736243    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-metrics-certs\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:37.202609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:37.202573    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:37.205133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:37.205111    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls\") pod \"maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:37.370075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:37.369573    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:37.802883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:37.802801    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"]
Nov 03 00:30:37.809502 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:30:37.803983    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b760644_262e_4470_b3a2_e462d3479d80.slice/crio-5c53729e6c04ba15fa3a3c47ff4f1eb22cf756b6a3f1866fb3c6914d25d9c002 WatchSource:0}: Error finding container 5c53729e6c04ba15fa3a3c47ff4f1eb22cf756b6a3f1866fb3c6914d25d9c002: Status 404 returned error can't find the container with id 5c53729e6c04ba15fa3a3c47ff4f1eb22cf756b6a3f1866fb3c6914d25d9c002
Nov 03 00:30:37.960998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:37.960961    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz" event={"ID":"2b760644-262e-4470-b3a2-e462d3479d80","Type":"ContainerStarted","Data":"5c53729e6c04ba15fa3a3c47ff4f1eb22cf756b6a3f1866fb3c6914d25d9c002"}
Nov 03 00:30:38.969311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:38.969228    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz" event={"ID":"2b760644-262e-4470-b3a2-e462d3479d80","Type":"ContainerStarted","Data":"aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948"}
Nov 03 00:30:38.969796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:38.969457    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:39.009640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:39.009182    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz" podStartSLOduration=2.657478296 podStartE2EDuration="3.009163262s" totalImagesPullingTime="351.684966ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:30:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:30:39.008603693 +0000 UTC m=+1476.563955744" watchObservedRunningTime="2026-11-03 00:30:39.009163262 +0000 UTC m=+1476.564515286"
Nov 03 00:30:40.098278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:40.093080    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"]
Nov 03 00:30:41.593282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:41.582221    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:30:45.020725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.019942    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:30:45.094116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.094076    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk" podUID="c4b7aba0-4c8d-4116-a042-1583b36a49f4" containerName="payload-processing" containerID="cri-o://6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982" gracePeriod=25
Nov 03 00:30:45.382210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.382180    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:30:45.495179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.495145    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4b7aba0-4c8d-4116-a042-1583b36a49f4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7aba0-4c8d-4116-a042-1583b36a49f4-plugins-config-volume\") pod \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") "
Nov 03 00:30:45.495387 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.495340    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4b7aba0-4c8d-4116-a042-1583b36a49f4-kube-api-access-6pfcp\" (UniqueName: \"kubernetes.io/projected/c4b7aba0-4c8d-4116-a042-1583b36a49f4-kube-api-access-6pfcp\") pod \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\" (UID: \"c4b7aba0-4c8d-4116-a042-1583b36a49f4\") "
Nov 03 00:30:45.495604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.495577    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4b7aba0-4c8d-4116-a042-1583b36a49f4-plugins-config-volume" pod "c4b7aba0-4c8d-4116-a042-1583b36a49f4" (UID: "c4b7aba0-4c8d-4116-a042-1583b36a49f4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:30:45.495933 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.495918    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4b7aba0-4c8d-4116-a042-1583b36a49f4-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:30:45.497567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.497547    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4b7aba0-4c8d-4116-a042-1583b36a49f4-kube-api-access-6pfcp" pod "c4b7aba0-4c8d-4116-a042-1583b36a49f4" (UID: "c4b7aba0-4c8d-4116-a042-1583b36a49f4"). InnerVolumeSpecName "kube-api-access-6pfcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:30:45.597930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:45.597846    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pfcp\" (UniqueName: \"kubernetes.io/projected/c4b7aba0-4c8d-4116-a042-1583b36a49f4-kube-api-access-6pfcp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:30:46.046851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.046819    6000 generic.go:356] "Generic (PLEG): container finished" podID="c4b7aba0-4c8d-4116-a042-1583b36a49f4" containerID="6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982" exitCode=0
Nov 03 00:30:46.047362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.046865    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk" event={"ID":"c4b7aba0-4c8d-4116-a042-1583b36a49f4","Type":"ContainerDied","Data":"6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982"}
Nov 03 00:30:46.047362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.046919    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk" event={"ID":"c4b7aba0-4c8d-4116-a042-1583b36a49f4","Type":"ContainerDied","Data":"5b90fa72643f04b15ab8d6fce8c2b17f009fb7139fb8ebafacda090f97d543cc"}
Nov 03 00:30:46.047362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.046868    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"
Nov 03 00:30:46.047362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.046949    6000 scope.go:122] "RemoveContainer" containerID="6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982"
Nov 03 00:30:46.057663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.057644    6000 scope.go:122] "RemoveContainer" containerID="6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982"
Nov 03 00:30:46.057938 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:30:46.057918    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982\": container with ID starting with 6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982 not found: ID does not exist" containerID="6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982"
Nov 03 00:30:46.057985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.057947    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982"} err="failed to get container status \"6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982\": rpc error: code = NotFound desc = could not find container \"6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982\": container with ID starting with 6b394a818e76eab79ae878b7c7a0f8e4fd9aa58e8ce9e6a3533d65f20009a982 not found: ID does not exist"
Nov 03 00:30:46.096326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.096298    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"]
Nov 03 00:30:46.107396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.107372    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ea275f-6465d6f9d7-hnvgk"]
Nov 03 00:30:46.568886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:30:46.568846    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4b7aba0-4c8d-4116-a042-1583b36a49f4" path="/var/lib/kubelet/pods/c4b7aba0-4c8d-4116-a042-1583b36a49f4/volumes"
Nov 03 00:30:49.897070 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:30:49.897029    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:31:02.324796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.324761    6000 generic.go:356] "Generic (PLEG): container finished" podID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerID="1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae" exitCode=0
Nov 03 00:31:02.325310 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.324835    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" event={"ID":"f097aed8-d75f-4dda-ac53-27eb137685a5","Type":"ContainerDied","Data":"1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae"}
Nov 03 00:31:02.325383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.325328    6000 scope.go:122] "RemoveContainer" containerID="1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae"
Nov 03 00:31:02.327104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.327079    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:31:02.327176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.327147    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:31:02.327213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.327178    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:31:02.662071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.662034    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"]
Nov 03 00:31:02.663133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.663095    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c4b7aba0-4c8d-4116-a042-1583b36a49f4" containerName="payload-processing"
Nov 03 00:31:02.663133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.663128    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4b7aba0-4c8d-4116-a042-1583b36a49f4" containerName="payload-processing"
Nov 03 00:31:02.663385 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.663311    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4b7aba0-4c8d-4116-a042-1583b36a49f4" containerName="payload-processing"
Nov 03 00:31:02.666580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.666556    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:02.685292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.685244    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-68738d\""
Nov 03 00:31:02.685552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.685516    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-68738d-dockercfg-29w4t\""
Nov 03 00:31:02.685961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.685937    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"]
Nov 03 00:31:02.833272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.822311    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnc6g\" (UniqueName: \"kubernetes.io/projected/4a616e87-408c-437c-94c5-bbbb7454d614-kube-api-access-wnc6g\") pod \"payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:02.833272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.822699    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a616e87-408c-437c-94c5-bbbb7454d614-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:02.833272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.825043    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"]
Nov 03 00:31:03.020533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.928233    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a616e87-408c-437c-94c5-bbbb7454d614-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:03.022181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.022149    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnc6g\" (UniqueName: \"kubernetes.io/projected/4a616e87-408c-437c-94c5-bbbb7454d614-kube-api-access-wnc6g\") pod \"payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:03.032959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.966384    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"]
Nov 03 00:31:03.046370 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.966468    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.046720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:02.938675    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a616e87-408c-437c-94c5-bbbb7454d614-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:03.084321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.079826    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnc6g\" (UniqueName: \"kubernetes.io/projected/4a616e87-408c-437c-94c5-bbbb7454d614-kube-api-access-wnc6g\") pod \"payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:03.134949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.134900    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs5t6\" (UniqueName: \"kubernetes.io/projected/3f5f82ec-61ba-456b-87b9-467f85256fd3-kube-api-access-gs5t6\") pod \"payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.135303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.135277    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5f82ec-61ba-456b-87b9-467f85256fd3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.236727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.236685    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs5t6\" (UniqueName: \"kubernetes.io/projected/3f5f82ec-61ba-456b-87b9-467f85256fd3-kube-api-access-gs5t6\") pod \"payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.238822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.238793    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5f82ec-61ba-456b-87b9-467f85256fd3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.241189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.239912    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5f82ec-61ba-456b-87b9-467f85256fd3-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.262115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.262081    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs5t6\" (UniqueName: \"kubernetes.io/projected/3f5f82ec-61ba-456b-87b9-467f85256fd3-kube-api-access-gs5t6\") pod \"payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.311411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.295879    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:03.336105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.336077    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" event={"ID":"f097aed8-d75f-4dda-ac53-27eb137685a5","Type":"ContainerStarted","Data":"4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4"}
Nov 03 00:31:03.420644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.420617    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:03.617736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.617702    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"]
Nov 03 00:31:03.620628 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:31:03.620600    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a616e87_408c_437c_94c5_bbbb7454d614.slice/crio-8388cfb8d63e083d4bc0fe75bdc59866f6b56506ad1c444417e81b6f8415edda WatchSource:0}: Error finding container 8388cfb8d63e083d4bc0fe75bdc59866f6b56506ad1c444417e81b6f8415edda: Status 404 returned error can't find the container with id 8388cfb8d63e083d4bc0fe75bdc59866f6b56506ad1c444417e81b6f8415edda
Nov 03 00:31:03.741994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:03.741968    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"]
Nov 03 00:31:03.744707 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:31:03.744680    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5f82ec_61ba_456b_87b9_467f85256fd3.slice/crio-cdec8d9d361bd450683754da274cf2676c36e677b79a34c95b829e068300ea2d WatchSource:0}: Error finding container cdec8d9d361bd450683754da274cf2676c36e677b79a34c95b829e068300ea2d: Status 404 returned error can't find the container with id cdec8d9d361bd450683754da274cf2676c36e677b79a34c95b829e068300ea2d
Nov 03 00:31:04.312282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.311380    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:31:04.313424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.313033    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:31:04.313424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.313092    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 00:31:04.352303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.349473    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m" event={"ID":"3f5f82ec-61ba-456b-87b9-467f85256fd3","Type":"ContainerStarted","Data":"db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb"}
Nov 03 00:31:04.352303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.349771    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m" event={"ID":"3f5f82ec-61ba-456b-87b9-467f85256fd3","Type":"ContainerStarted","Data":"cdec8d9d361bd450683754da274cf2676c36e677b79a34c95b829e068300ea2d"}
Nov 03 00:31:04.352303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.349805    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:04.352303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.350944    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb" event={"ID":"4a616e87-408c-437c-94c5-bbbb7454d614","Type":"ContainerStarted","Data":"44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1"}
Nov 03 00:31:04.352303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.350988    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb" event={"ID":"4a616e87-408c-437c-94c5-bbbb7454d614","Type":"ContainerStarted","Data":"8388cfb8d63e083d4bc0fe75bdc59866f6b56506ad1c444417e81b6f8415edda"}
Nov 03 00:31:04.352303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.351099    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:04.418377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.415590    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m" podStartSLOduration=2.414286916 podStartE2EDuration="2.414286916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:04.408728919 +0000 UTC m=+1501.964080944" watchObservedRunningTime="2026-11-03 00:31:04.414286916 +0000 UTC m=+1501.969638940"
Nov 03 00:31:04.447398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:04.447345    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb" podStartSLOduration=2.447330823 podStartE2EDuration="2.447330823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:04.443640385 +0000 UTC m=+1501.998992410" watchObservedRunningTime="2026-11-03 00:31:04.447330823 +0000 UTC m=+1502.002682890"
Nov 03 00:31:05.304289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:05.303771    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:31:05.304289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:05.303846    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 00:31:06.305285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:06.301314    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:31:06.305285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:06.301375    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 00:31:07.295279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:07.290698    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:31:07.295279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:07.290768    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 00:31:07.359284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:07.358392    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:31:08.297281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:08.289651    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 03 00:31:08.297281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:08.289724    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 03 00:31:09.292880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:09.292847    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:31:09.294084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:09.294065    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:31:11.958774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:11.958740    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:31:14.582002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:14.581972    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"]
Nov 03 00:31:14.616106 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:14.616077    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"]
Nov 03 00:31:15.388609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:15.388578    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:31:15.389499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:15.389473    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:31:19.582637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.582599    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt" podUID="2893fb17-1482-4e46-ae4d-01355e048194" containerName="payload-processing" containerID="cri-o://d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba" gracePeriod=25
Nov 03 00:31:19.617142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.617090    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r" podUID="3f0964cb-09f8-4208-a29d-0162bc764ac1" containerName="payload-pre-processing" containerID="cri-o://1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d" gracePeriod=25
Nov 03 00:31:19.931810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.931778    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:31:19.936566 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:31:19.936534    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:31:19.988997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.988969    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2893fb17-1482-4e46-ae4d-01355e048194-kube-api-access-86hrk\" (UniqueName: \"kubernetes.io/projected/2893fb17-1482-4e46-ae4d-01355e048194-kube-api-access-86hrk\") pod \"2893fb17-1482-4e46-ae4d-01355e048194\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") "
Nov 03 00:31:19.989166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.989098    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2893fb17-1482-4e46-ae4d-01355e048194-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2893fb17-1482-4e46-ae4d-01355e048194-plugins-config-volume\") pod \"2893fb17-1482-4e46-ae4d-01355e048194\" (UID: \"2893fb17-1482-4e46-ae4d-01355e048194\") "
Nov 03 00:31:19.989664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.989640    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2893fb17-1482-4e46-ae4d-01355e048194-plugins-config-volume" pod "2893fb17-1482-4e46-ae4d-01355e048194" (UID: "2893fb17-1482-4e46-ae4d-01355e048194"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:31:19.994999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.994484    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2893fb17-1482-4e46-ae4d-01355e048194-kube-api-access-86hrk" pod "2893fb17-1482-4e46-ae4d-01355e048194" (UID: "2893fb17-1482-4e46-ae4d-01355e048194"). InnerVolumeSpecName "kube-api-access-86hrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:31:19.997316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:19.997294    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:31:20.090767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.090733    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f0964cb-09f8-4208-a29d-0162bc764ac1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0964cb-09f8-4208-a29d-0162bc764ac1-plugins-config-volume\") pod \"3f0964cb-09f8-4208-a29d-0162bc764ac1\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") "
Nov 03 00:31:20.090945 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.090864    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f0964cb-09f8-4208-a29d-0162bc764ac1-kube-api-access-bq8fw\" (UniqueName: \"kubernetes.io/projected/3f0964cb-09f8-4208-a29d-0162bc764ac1-kube-api-access-bq8fw\") pod \"3f0964cb-09f8-4208-a29d-0162bc764ac1\" (UID: \"3f0964cb-09f8-4208-a29d-0162bc764ac1\") "
Nov 03 00:31:20.091138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.091113    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f0964cb-09f8-4208-a29d-0162bc764ac1-plugins-config-volume" pod "3f0964cb-09f8-4208-a29d-0162bc764ac1" (UID: "3f0964cb-09f8-4208-a29d-0162bc764ac1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:31:20.091669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.091599    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86hrk\" (UniqueName: \"kubernetes.io/projected/2893fb17-1482-4e46-ae4d-01355e048194-kube-api-access-86hrk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:20.091669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.091626    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2893fb17-1482-4e46-ae4d-01355e048194-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:20.091669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.091646    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f0964cb-09f8-4208-a29d-0162bc764ac1-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:20.093331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.093305    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f0964cb-09f8-4208-a29d-0162bc764ac1-kube-api-access-bq8fw" pod "3f0964cb-09f8-4208-a29d-0162bc764ac1" (UID: "3f0964cb-09f8-4208-a29d-0162bc764ac1"). InnerVolumeSpecName "kube-api-access-bq8fw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:31:20.193402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.193326    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bq8fw\" (UniqueName: \"kubernetes.io/projected/3f0964cb-09f8-4208-a29d-0162bc764ac1-kube-api-access-bq8fw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:20.539520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.539432    6000 generic.go:356] "Generic (PLEG): container finished" podID="3f0964cb-09f8-4208-a29d-0162bc764ac1" containerID="1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d" exitCode=0
Nov 03 00:31:20.539520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.539492    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"
Nov 03 00:31:20.539520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.539510    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r" event={"ID":"3f0964cb-09f8-4208-a29d-0162bc764ac1","Type":"ContainerDied","Data":"1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d"}
Nov 03 00:31:20.539807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.539578    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r" event={"ID":"3f0964cb-09f8-4208-a29d-0162bc764ac1","Type":"ContainerDied","Data":"60546b738d016dd959dd3abca451c5330b41211cf50ef05aecee7791960a235d"}
Nov 03 00:31:20.539807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.539602    6000 scope.go:122] "RemoveContainer" containerID="1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d"
Nov 03 00:31:20.541132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.541110    6000 generic.go:356] "Generic (PLEG): container finished" podID="2893fb17-1482-4e46-ae4d-01355e048194" containerID="d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba" exitCode=0
Nov 03 00:31:20.541275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.541174    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"
Nov 03 00:31:20.541275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.541190    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt" event={"ID":"2893fb17-1482-4e46-ae4d-01355e048194","Type":"ContainerDied","Data":"d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba"}
Nov 03 00:31:20.541275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.541230    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt" event={"ID":"2893fb17-1482-4e46-ae4d-01355e048194","Type":"ContainerDied","Data":"4c1175b7695e4e5ae8839b166d1e83030a5cf6ec8500b7efb640754c8fdcb9ca"}
Nov 03 00:31:20.550390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.550371    6000 scope.go:122] "RemoveContainer" containerID="1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d"
Nov 03 00:31:20.550713 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:31:20.550691    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d\": container with ID starting with 1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d not found: ID does not exist" containerID="1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d"
Nov 03 00:31:20.550815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.550720    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d"} err="failed to get container status \"1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d\": rpc error: code = NotFound desc = could not find container \"1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d\": container with ID starting with 1c3c79fb81328b6eb60b8aa53bcb1f68778f07ec5db7e31e4a0c4d9b99dc180d not found: ID does not exist"
Nov 03 00:31:20.550815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.550743    6000 scope.go:122] "RemoveContainer" containerID="d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba"
Nov 03 00:31:20.560129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.560113    6000 scope.go:122] "RemoveContainer" containerID="d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba"
Nov 03 00:31:20.560460 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:31:20.560438    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba\": container with ID starting with d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba not found: ID does not exist" containerID="d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba"
Nov 03 00:31:20.560520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.560467    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba"} err="failed to get container status \"d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba\": rpc error: code = NotFound desc = could not find container \"d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba\": container with ID starting with d9bb45d370677971c905d13e034f19e0eaaf9c581bdab2a782a9962fb524deba not found: ID does not exist"
Nov 03 00:31:20.590283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.590236    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"]
Nov 03 00:31:20.606814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.606786    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-b6cd31-695659fd9f-hw46r"]
Nov 03 00:31:20.637952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.637921    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"]
Nov 03 00:31:20.641174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:20.641150    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-b6cd31-59d6b865df-bvvgt"]
Nov 03 00:31:22.568019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:22.567983    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2893fb17-1482-4e46-ae4d-01355e048194" path="/var/lib/kubelet/pods/2893fb17-1482-4e46-ae4d-01355e048194/volumes"
Nov 03 00:31:22.568665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:22.568642    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f0964cb-09f8-4208-a29d-0162bc764ac1" path="/var/lib/kubelet/pods/3f0964cb-09f8-4208-a29d-0162bc764ac1/volumes"
Nov 03 00:31:31.103182 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.103150    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"]
Nov 03 00:31:31.103910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.103895    6000 cpu_manager.go:414] "containerMap: removing container" podUID="3f0964cb-09f8-4208-a29d-0162bc764ac1" containerName="payload-pre-processing"
Nov 03 00:31:31.103964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.103912    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f0964cb-09f8-4208-a29d-0162bc764ac1" containerName="payload-pre-processing"
Nov 03 00:31:31.104001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.103964    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2893fb17-1482-4e46-ae4d-01355e048194" containerName="payload-processing"
Nov 03 00:31:31.104001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.103971    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2893fb17-1482-4e46-ae4d-01355e048194" containerName="payload-processing"
Nov 03 00:31:31.104082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.104073    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f0964cb-09f8-4208-a29d-0162bc764ac1" containerName="payload-pre-processing"
Nov 03 00:31:31.104116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.104093    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2893fb17-1482-4e46-ae4d-01355e048194" containerName="payload-processing"
Nov 03 00:31:31.107472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.107454    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.128696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.128667    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"]
Nov 03 00:31:31.135845 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.135824    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-b-68738d-serving-cert\""
Nov 03 00:31:31.146981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.146947    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-maas-api-tls\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.147156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.147004    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7z86\" (UniqueName: \"kubernetes.io/projected/cd3dc0fe-bd42-43fa-976f-404b5dad83db-kube-api-access-v7z86\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.147239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.147155    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-metrics-certs\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.147355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.147299    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3dc0fe-bd42-43fa-976f-404b5dad83db-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.248884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.248852    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-maas-api-tls\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.249101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.248895    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7z86\" (UniqueName: \"kubernetes.io/projected/cd3dc0fe-bd42-43fa-976f-404b5dad83db-kube-api-access-v7z86\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.249101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.249020    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-metrics-certs\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.249101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.249088    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3dc0fe-bd42-43fa-976f-404b5dad83db-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.249872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.249812    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3dc0fe-bd42-43fa-976f-404b5dad83db-openshift-service-ca\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.251719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.251697    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-maas-api-tls\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.251810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.251752    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-metrics-certs\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.268176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.268147    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7z86\" (UniqueName: \"kubernetes.io/projected/cd3dc0fe-bd42-43fa-976f-404b5dad83db-kube-api-access-v7z86\") pod \"maas-api-e2e-praxis-b-68738d-cf474d67-q6chz\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.423306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.423151    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:31.742540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:31.742515    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"]
Nov 03 00:31:31.742824 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:31:31.742803    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3dc0fe_bd42_43fa_976f_404b5dad83db.slice/crio-7745b7a0b67011deaa337db14c4769503381a719284d7d9ae4c7edf264d20091 WatchSource:0}: Error finding container 7745b7a0b67011deaa337db14c4769503381a719284d7d9ae4c7edf264d20091: Status 404 returned error can't find the container with id 7745b7a0b67011deaa337db14c4769503381a719284d7d9ae4c7edf264d20091
Nov 03 00:31:32.639801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:32.639756    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz" event={"ID":"cd3dc0fe-bd42-43fa-976f-404b5dad83db","Type":"ContainerStarted","Data":"3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126"}
Nov 03 00:31:32.640279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:32.639818    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz" event={"ID":"cd3dc0fe-bd42-43fa-976f-404b5dad83db","Type":"ContainerStarted","Data":"7745b7a0b67011deaa337db14c4769503381a719284d7d9ae4c7edf264d20091"}
Nov 03 00:31:32.640279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:32.639914    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:32.678741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:32.678664    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz" podStartSLOduration=1.315195544 podStartE2EDuration="1.678643461s" totalImagesPullingTime="363.447917ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:31:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:31:32.676885299 +0000 UTC m=+1530.232237323" watchObservedRunningTime="2026-11-03 00:31:32.678643461 +0000 UTC m=+1530.233995485"
Nov 03 00:31:34.304452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.304417    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"]
Nov 03 00:31:34.305286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.305238    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx" podUID="5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" containerName="maas-api" containerID="cri-o://7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1" gracePeriod=30
Nov 03 00:31:34.609134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.609107    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:31:34.656594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.656542    6000 generic.go:356] "Generic (PLEG): container finished" podID="5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" containerID="7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1" exitCode=0
Nov 03 00:31:34.656842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.656621    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"
Nov 03 00:31:34.656842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.656638    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx" event={"ID":"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe","Type":"ContainerDied","Data":"7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1"}
Nov 03 00:31:34.656842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.656689    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx" event={"ID":"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe","Type":"ContainerDied","Data":"682af6a36b33726cd8f99cac6e195d70be0aba72fa611530e9340ccee5724b5e"}
Nov 03 00:31:34.656842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.656719    6000 scope.go:122] "RemoveContainer" containerID="7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1"
Nov 03 00:31:34.670083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.670060    6000 scope.go:122] "RemoveContainer" containerID="7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1"
Nov 03 00:31:34.670504 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:31:34.670476    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1\": container with ID starting with 7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1 not found: ID does not exist" containerID="7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1"
Nov 03 00:31:34.670625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.670517    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1"} err="failed to get container status \"7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1\": rpc error: code = NotFound desc = could not find container \"7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1\": container with ID starting with 7c72098809076e0d5b905c4238474c927dcefe208efbad10c4172f8c52243ec1 not found: ID does not exist"
Nov 03 00:31:34.681189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.681164    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-openshift-service-ca\") pod \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") "
Nov 03 00:31:34.681332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.681276    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-metrics-certs\") pod \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") "
Nov 03 00:31:34.681409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.681363    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-kube-api-access-jdbst\" (UniqueName: \"kubernetes.io/projected/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-kube-api-access-jdbst\") pod \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") "
Nov 03 00:31:34.681409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.681391    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-maas-api-tls\") pod \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\" (UID: \"5f8a9c80-0b03-4ac6-9dab-97fb3da726fe\") "
Nov 03 00:31:34.681609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.681581    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-openshift-service-ca" pod "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" (UID: "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:31:34.682038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.682010    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:34.683932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.683905    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-maas-api-tls" pod "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" (UID: "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:31:34.684314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.684293    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-metrics-certs" pod "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" (UID: "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:31:34.684531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.684510    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-kube-api-access-jdbst" pod "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" (UID: "5f8a9c80-0b03-4ac6-9dab-97fb3da726fe"). InnerVolumeSpecName "kube-api-access-jdbst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:31:34.795432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.795369    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdbst\" (UniqueName: \"kubernetes.io/projected/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-kube-api-access-jdbst\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:34.795719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.795685    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:34.795841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:34.795829    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:35.091572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:35.091540    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"]
Nov 03 00:31:35.103018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:35.102992    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-54f6d8-d78869f-n82xx"]
Nov 03 00:31:36.569034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:36.569000    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" path="/var/lib/kubelet/pods/5f8a9c80-0b03-4ac6-9dab-97fb3da726fe/volumes"
Nov 03 00:31:38.652959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:38.652928    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:31:45.409624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:45.409530    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"]
Nov 03 00:31:45.410137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:45.409806    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy" containerID="cri-o://dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357" gracePeriod=30
Nov 03 00:31:49.873406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:49.873357    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:31:49.979151 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:31:49.979110    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:31:50.734828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.734415    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:31:50.873682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.873646    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istiod-ca-cert\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.873790    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-data\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.873832    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-credential-socket\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.873892    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-certs\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.873966    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-envoy\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.874071    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-kube-api-access-kbdgd\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-kube-api-access-kbdgd\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.874119    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-token\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-token\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.874189    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-socket\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.874480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.874222    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-podinfo\") pod \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\" (UID: \"ef1b4163-a397-484b-973f-a4d8fb9f6e88\") "
Nov 03 00:31:50.875578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.875531    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-data" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:31:50.875718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.875641    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istiod-ca-cert" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:31:50.875927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.875905    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-socket" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:31:50.876069 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.876012    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-credential-socket" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:31:50.876144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.876101    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-certs" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:31:50.893453 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.893415    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-envoy" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:31:50.915788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.915268    6000 generic.go:356] "Generic (PLEG): container finished" podID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerID="dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357" exitCode=0
Nov 03 00:31:50.915788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.915561    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" event={"ID":"ef1b4163-a397-484b-973f-a4d8fb9f6e88","Type":"ContainerDied","Data":"dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357"}
Nov 03 00:31:50.915788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.915594    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"
Nov 03 00:31:50.915788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.915621    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp" event={"ID":"ef1b4163-a397-484b-973f-a4d8fb9f6e88","Type":"ContainerDied","Data":"36d1309f6cf72e05a5163a6f506d991b81ce31ce6b18fa0a26a04415d7e1ddec"}
Nov 03 00:31:50.915788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.915654    6000 scope.go:122] "RemoveContainer" containerID="dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357"
Nov 03 00:31:50.930842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.929704    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-kube-api-access-kbdgd" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "kube-api-access-kbdgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:31:50.930842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.930462    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-token" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:31:50.930842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.930733    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-podinfo" pod "ef1b4163-a397-484b-973f-a4d8fb9f6e88" (UID: "ef1b4163-a397-484b-973f-a4d8fb9f6e88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:31:50.949073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.949040    6000 scope.go:122] "RemoveContainer" containerID="752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97"
Nov 03 00:31:50.960208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.960178    6000 scope.go:122] "RemoveContainer" containerID="dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357"
Nov 03 00:31:50.960556 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:31:50.960529    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357\": container with ID starting with dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357 not found: ID does not exist" containerID="dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357"
Nov 03 00:31:50.960661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.960571    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357"} err="failed to get container status \"dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357\": rpc error: code = NotFound desc = could not find container \"dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357\": container with ID starting with dfad41f0d327de5d9075e6b7343426e4f0e00d22956e930473ee8bf268f6b357 not found: ID does not exist"
Nov 03 00:31:50.960661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.960599    6000 scope.go:122] "RemoveContainer" containerID="752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97"
Nov 03 00:31:50.960952 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:31:50.960927    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97\": container with ID starting with 752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97 not found: ID does not exist" containerID="752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97"
Nov 03 00:31:50.961009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.960964    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97"} err="failed to get container status \"752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97\": rpc error: code = NotFound desc = could not find container \"752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97\": container with ID starting with 752b17abf1c43aa5553a5ee21027062b2772abd615a09f8d10ce8ee5013a9d97 not found: ID does not exist"
Nov 03 00:31:50.975903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975870    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbdgd\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-kube-api-access-kbdgd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.975903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975898    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.976082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975913    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.976082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975924    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.976082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975936    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.976082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975946    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.976082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975958    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.976082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975969    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:50.976082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:50.975981    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef1b4163-a397-484b-973f-a4d8fb9f6e88-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:31:51.264896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:51.264860    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"]
Nov 03 00:31:51.278104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:51.278072    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-54f6d8-openshift-default-7b6b7d6d89-5kfjp"]
Nov 03 00:31:52.572429 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:31:52.572400    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" path="/var/lib/kubelet/pods/ef1b4163-a397-484b-973f-a4d8fb9f6e88/volumes"
Nov 03 00:32:05.236479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:05.236444    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"]
Nov 03 00:32:05.237007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:05.236695    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy" containerID="cri-o://bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84" gracePeriod=30
Nov 03 00:32:05.251397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:05.251361    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"]
Nov 03 00:32:05.251654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:05.251607    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy" containerID="cri-o://16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180" gracePeriod=30
Nov 03 00:32:05.345308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:05.345235    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"]
Nov 03 00:32:05.346471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:05.346146    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy" containerID="cri-o://0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf" gracePeriod=30
Nov 03 00:32:09.516173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:09.516139    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"]
Nov 03 00:32:09.516640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:09.516361    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf" podUID="1f1ddfe4-135a-4efc-af63-ba784dabca6a" containerName="maas-api" containerID="cri-o://fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b" gracePeriod=30
Nov 03 00:32:09.835123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:09.835096    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:32:10.027282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.018759    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f1ddfe4-135a-4efc-af63-ba784dabca6a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1ddfe4-135a-4efc-af63-ba784dabca6a-openshift-service-ca\") pod \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") "
Nov 03 00:32:10.027282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.018820    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-maas-api-tls\") pod \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") "
Nov 03 00:32:10.027282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.019086    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f1ddfe4-135a-4efc-af63-ba784dabca6a-kube-api-access-z98qj\" (UniqueName: \"kubernetes.io/projected/1f1ddfe4-135a-4efc-af63-ba784dabca6a-kube-api-access-z98qj\") pod \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") "
Nov 03 00:32:10.027282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.019309    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-metrics-certs\") pod \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\" (UID: \"1f1ddfe4-135a-4efc-af63-ba784dabca6a\") "
Nov 03 00:32:10.045775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.039506    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f1ddfe4-135a-4efc-af63-ba784dabca6a-openshift-service-ca" pod "1f1ddfe4-135a-4efc-af63-ba784dabca6a" (UID: "1f1ddfe4-135a-4efc-af63-ba784dabca6a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.063635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.055446    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-metrics-certs" pod "1f1ddfe4-135a-4efc-af63-ba784dabca6a" (UID: "1f1ddfe4-135a-4efc-af63-ba784dabca6a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:10.063635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.055713    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f1ddfe4-135a-4efc-af63-ba784dabca6a-kube-api-access-z98qj" pod "1f1ddfe4-135a-4efc-af63-ba784dabca6a" (UID: "1f1ddfe4-135a-4efc-af63-ba784dabca6a"). InnerVolumeSpecName "kube-api-access-z98qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.079278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.076529    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-maas-api-tls" pod "1f1ddfe4-135a-4efc-af63-ba784dabca6a" (UID: "1f1ddfe4-135a-4efc-af63-ba784dabca6a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:10.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.113451    6000 generic.go:356] "Generic (PLEG): container finished" podID="1f1ddfe4-135a-4efc-af63-ba784dabca6a" containerID="fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b" exitCode=0
Nov 03 00:32:10.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.113565    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf" event={"ID":"1f1ddfe4-135a-4efc-af63-ba784dabca6a","Type":"ContainerDied","Data":"fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b"}
Nov 03 00:32:10.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.113618    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf" event={"ID":"1f1ddfe4-135a-4efc-af63-ba784dabca6a","Type":"ContainerDied","Data":"22dbac9e60435e56db55364d97aee18699883340ffe240f3de14323f9bfc4cc4"}
Nov 03 00:32:10.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.113649    6000 scope.go:122] "RemoveContainer" containerID="fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b"
Nov 03 00:32:10.117280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.113820    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"
Nov 03 00:32:10.134284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.127073    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z98qj\" (UniqueName: \"kubernetes.io/projected/1f1ddfe4-135a-4efc-af63-ba784dabca6a-kube-api-access-z98qj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.134284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.127113    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.134284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.127130    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f1ddfe4-135a-4efc-af63-ba784dabca6a-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.134284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.127144    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f1ddfe4-135a-4efc-af63-ba784dabca6a-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.260020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.259632    6000 scope.go:122] "RemoveContainer" containerID="fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b"
Nov 03 00:32:10.260815 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:10.260715    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b\": container with ID starting with fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b not found: ID does not exist" containerID="fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b"
Nov 03 00:32:10.260815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.260756    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b"} err="failed to get container status \"fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b\": rpc error: code = NotFound desc = could not find container \"fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b\": container with ID starting with fd4ce4e28370dd6bdd31fac6d601b10143114f87eaf24a0856c6da9358fe2f3b not found: ID does not exist"
Nov 03 00:32:10.261855 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.261825    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"]
Nov 03 00:32:10.280928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.280753    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-846492-7f5b4556f5-gs4hf"]
Nov 03 00:32:10.525282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.525034    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.643757    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ba0b79-946a-44df-b0b3-befc1d76751f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ba0b79-946a-44df-b0b3-befc1d76751f-istiod-ca-cert\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.643817    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-podinfo\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.643859    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-credential-socket\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.644052    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-certs\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.644192    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-socket\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.644289    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-data\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.644362    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-token\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-token\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.644428    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-envoy\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.644505    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-kube-api-access-gmm6h\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-kube-api-access-gmm6h\") pod \"91ba0b79-946a-44df-b0b3-befc1d76751f\" (UID: \"91ba0b79-946a-44df-b0b3-befc1d76751f\") "
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.660436    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-certs" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.661202    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ba0b79-946a-44df-b0b3-befc1d76751f-istiod-ca-cert" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.663009    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-credential-socket" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.663192    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-data" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.673383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.663370    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-socket" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.685436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.685370    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f1ddfe4-135a-4efc-af63-ba784dabca6a" path="/var/lib/kubelet/pods/1f1ddfe4-135a-4efc-af63-ba784dabca6a/volumes"
Nov 03 00:32:10.710300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.709085    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-kube-api-access-gmm6h" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "kube-api-access-gmm6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.711496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.711461    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-envoy" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.713401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.713372    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-podinfo" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.714446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.714416    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-token" pod "91ba0b79-946a-44df-b0b3-befc1d76751f" (UID: "91ba0b79-946a-44df-b0b3-befc1d76751f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.747241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747188    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747227    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmm6h\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-kube-api-access-gmm6h\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747242    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91ba0b79-946a-44df-b0b3-befc1d76751f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747292    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747306    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747322    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747336    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747348    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.747502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.747361    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91ba0b79-946a-44df-b0b3-befc1d76751f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.764391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.764334    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:32:10.775178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.775151    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:32:10.848720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.848682    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-kube-api-access-dbvvh\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-kube-api-access-dbvvh\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.848879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.848766    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-socket\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.848879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.848868    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-podinfo\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.848989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.848901    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-envoy\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.848989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.848984    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-data\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.849088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.849043    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-certs\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.849142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.849103    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-credential-socket\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.849200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.849144    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istiod-ca-cert\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.849200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.849183    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-token\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-token\") pod \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\" (UID: \"3101c60a-bbf0-41d3-99cf-4f81bff4bff8\") "
Nov 03 00:32:10.849793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.849749    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-data" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.849906 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.849791    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-credential-socket" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.850175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.850096    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istiod-ca-cert" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.850392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.850249    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.850479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.850404    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.850479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.850419    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.850593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.850486    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-certs" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.851408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.851364    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-socket" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.852508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.852447    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-podinfo" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.852739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.852715    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-kube-api-access-dbvvh" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "kube-api-access-dbvvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.852903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.852886    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-token" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.853962 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.853939    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-envoy" pod "3101c60a-bbf0-41d3-99cf-4f81bff4bff8" (UID: "3101c60a-bbf0-41d3-99cf-4f81bff4bff8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.951077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.950991    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-certs\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951103    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-socket\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951208    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-data\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951239    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istiod-ca-cert\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951246    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-certs" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.951547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951305    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-envoy\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951372    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-podinfo\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951418    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-socket" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.951547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951480    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-credential-socket\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951482    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-data" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.951547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951516    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-kube-api-access-h85j8\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-kube-api-access-h85j8\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951559    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-token\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-token\") pod \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\" (UID: \"1c43838a-640d-4fe9-81c6-f1cb1d8eb327\") "
Nov 03 00:32:10.951838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951568    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istiod-ca-cert" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:10.951838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.951732    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-credential-socket" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.952213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952191    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952219    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952232    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952243    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952294    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952309    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvvh\" (UniqueName: \"kubernetes.io/projected/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-kube-api-access-dbvvh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952322    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952334    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952347    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952358    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.952669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.952372    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3101c60a-bbf0-41d3-99cf-4f81bff4bff8-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:10.953907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.953881    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-envoy" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:10.954029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.953938    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-kube-api-access-h85j8" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "kube-api-access-h85j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:10.954299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.954240    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-podinfo" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:10.954398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:10.954300    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-token" pod "1c43838a-640d-4fe9-81c6-f1cb1d8eb327" (UID: "1c43838a-640d-4fe9-81c6-f1cb1d8eb327"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:11.053754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.053720    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.053754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.053748    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.053754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.053757    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h85j8\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-kube-api-access-h85j8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.053754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.053766    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1c43838a-640d-4fe9-81c6-f1cb1d8eb327-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:11.122593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.122557    6000 generic.go:356] "Generic (PLEG): container finished" podID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerID="bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84" exitCode=0
Nov 03 00:32:11.122790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.122613    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"
Nov 03 00:32:11.122790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.122643    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" event={"ID":"91ba0b79-946a-44df-b0b3-befc1d76751f","Type":"ContainerDied","Data":"bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84"}
Nov 03 00:32:11.122790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.122707    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj" event={"ID":"91ba0b79-946a-44df-b0b3-befc1d76751f","Type":"ContainerDied","Data":"b59afc8251fb91ab4decc893bc3570ea693f71e959625c35cafc55abfd781d59"}
Nov 03 00:32:11.122790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.122736    6000 scope.go:122] "RemoveContainer" containerID="bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84"
Nov 03 00:32:11.124142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.124120    6000 generic.go:356] "Generic (PLEG): container finished" podID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerID="0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf" exitCode=0
Nov 03 00:32:11.124302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.124162    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"
Nov 03 00:32:11.124302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.124205    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" event={"ID":"1c43838a-640d-4fe9-81c6-f1cb1d8eb327","Type":"ContainerDied","Data":"0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf"}
Nov 03 00:32:11.124438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.124249    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk" event={"ID":"1c43838a-640d-4fe9-81c6-f1cb1d8eb327","Type":"ContainerDied","Data":"8e247e40a3f82de84f7d6c9d037f38096132a21692c50d39144174e2686bfcf9"}
Nov 03 00:32:11.125829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.125806    6000 generic.go:356] "Generic (PLEG): container finished" podID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerID="16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180" exitCode=0
Nov 03 00:32:11.125955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.125840    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"
Nov 03 00:32:11.125955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.125884    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" event={"ID":"3101c60a-bbf0-41d3-99cf-4f81bff4bff8","Type":"ContainerDied","Data":"16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180"}
Nov 03 00:32:11.125955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.125936    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n" event={"ID":"3101c60a-bbf0-41d3-99cf-4f81bff4bff8","Type":"ContainerDied","Data":"e10c728252853d7e54d59425248d24aab2a9fc2c7aeed3d0f4c4109dd11dbe85"}
Nov 03 00:32:11.134856 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.134828    6000 scope.go:122] "RemoveContainer" containerID="bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84"
Nov 03 00:32:11.135233 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:11.135134    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84\": container with ID starting with bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84 not found: ID does not exist" containerID="bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84"
Nov 03 00:32:11.135233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.135168    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84"} err="failed to get container status \"bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84\": rpc error: code = NotFound desc = could not find container \"bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84\": container with ID starting with bffe7897a71c0c88447343eeef2f27a25358d6d2ec445a3822f3fc59583ebc84 not found: ID does not exist"
Nov 03 00:32:11.135233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.135190    6000 scope.go:122] "RemoveContainer" containerID="0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf"
Nov 03 00:32:11.145009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.144994    6000 scope.go:122] "RemoveContainer" containerID="0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf"
Nov 03 00:32:11.145457 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:11.145245    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf\": container with ID starting with 0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf not found: ID does not exist" containerID="0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf"
Nov 03 00:32:11.145457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.145284    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf"} err="failed to get container status \"0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf\": rpc error: code = NotFound desc = could not find container \"0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf\": container with ID starting with 0bdf936696652ba656b8ec678784d7b14d88fd787b13e4b804335366cebdb4cf not found: ID does not exist"
Nov 03 00:32:11.145457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.145301    6000 scope.go:122] "RemoveContainer" containerID="16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180"
Nov 03 00:32:11.154907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.154881    6000 scope.go:122] "RemoveContainer" containerID="16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180"
Nov 03 00:32:11.155189 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:11.155164    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180\": container with ID starting with 16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180 not found: ID does not exist" containerID="16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180"
Nov 03 00:32:11.155277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.155198    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180"} err="failed to get container status \"16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180\": rpc error: code = NotFound desc = could not find container \"16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180\": container with ID starting with 16d3264fabf9880b184cb934cd4adc2da437ddcb513ebc90e2ab7249c0631180 not found: ID does not exist"
Nov 03 00:32:11.190377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.190348    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"]
Nov 03 00:32:11.201685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.201626    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-4jnzj"]
Nov 03 00:32:11.245896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.245873    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"]
Nov 03 00:32:11.253315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.253168    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-n4k2n"]
Nov 03 00:32:11.281767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.281732    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"]
Nov 03 00:32:11.286298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:11.286249    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-gb6pk"]
Nov 03 00:32:12.568795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:12.568762    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" path="/var/lib/kubelet/pods/1c43838a-640d-4fe9-81c6-f1cb1d8eb327/volumes"
Nov 03 00:32:12.569294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:12.569280    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" path="/var/lib/kubelet/pods/3101c60a-bbf0-41d3-99cf-4f81bff4bff8/volumes"
Nov 03 00:32:12.569734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:12.569721    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" path="/var/lib/kubelet/pods/91ba0b79-946a-44df-b0b3-befc1d76751f/volumes"
Nov 03 00:32:16.476559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.476519    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"]
Nov 03 00:32:16.477087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.476797    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw" podUID="de43d331-7219-4337-a7b4-dcb90588e10f" containerName="maas-api" containerID="cri-o://e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0" gracePeriod=30
Nov 03 00:32:16.899280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.892964    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:32:16.994281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.993002    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de43d331-7219-4337-a7b4-dcb90588e10f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de43d331-7219-4337-a7b4-dcb90588e10f-openshift-service-ca\") pod \"de43d331-7219-4337-a7b4-dcb90588e10f\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") "
Nov 03 00:32:16.994281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.993095    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-maas-api-tls\") pod \"de43d331-7219-4337-a7b4-dcb90588e10f\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") "
Nov 03 00:32:16.994281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.993966    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de43d331-7219-4337-a7b4-dcb90588e10f-openshift-service-ca" pod "de43d331-7219-4337-a7b4-dcb90588e10f" (UID: "de43d331-7219-4337-a7b4-dcb90588e10f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:16.994281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.994088    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de43d331-7219-4337-a7b4-dcb90588e10f-kube-api-access-k2h72\" (UniqueName: \"kubernetes.io/projected/de43d331-7219-4337-a7b4-dcb90588e10f-kube-api-access-k2h72\") pod \"de43d331-7219-4337-a7b4-dcb90588e10f\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") "
Nov 03 00:32:16.994281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.994173    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-metrics-certs\") pod \"de43d331-7219-4337-a7b4-dcb90588e10f\" (UID: \"de43d331-7219-4337-a7b4-dcb90588e10f\") "
Nov 03 00:32:16.998026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.995642    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de43d331-7219-4337-a7b4-dcb90588e10f-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:17.002279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:16.998538    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-metrics-certs" pod "de43d331-7219-4337-a7b4-dcb90588e10f" (UID: "de43d331-7219-4337-a7b4-dcb90588e10f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:17.030971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.030901    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-maas-api-tls" pod "de43d331-7219-4337-a7b4-dcb90588e10f" (UID: "de43d331-7219-4337-a7b4-dcb90588e10f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:17.031363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.031312    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de43d331-7219-4337-a7b4-dcb90588e10f-kube-api-access-k2h72" pod "de43d331-7219-4337-a7b4-dcb90588e10f" (UID: "de43d331-7219-4337-a7b4-dcb90588e10f"). InnerVolumeSpecName "kube-api-access-k2h72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:17.097585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.097496    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:17.097585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.097526    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2h72\" (UniqueName: \"kubernetes.io/projected/de43d331-7219-4337-a7b4-dcb90588e10f-kube-api-access-k2h72\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:17.097585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.097536    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de43d331-7219-4337-a7b4-dcb90588e10f-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:17.185655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.185623    6000 generic.go:356] "Generic (PLEG): container finished" podID="de43d331-7219-4337-a7b4-dcb90588e10f" containerID="e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0" exitCode=0
Nov 03 00:32:17.185829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.185682    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"
Nov 03 00:32:17.185829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.185705    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw" event={"ID":"de43d331-7219-4337-a7b4-dcb90588e10f","Type":"ContainerDied","Data":"e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0"}
Nov 03 00:32:17.185829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.185761    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw" event={"ID":"de43d331-7219-4337-a7b4-dcb90588e10f","Type":"ContainerDied","Data":"2ccebfddf1222adf7fb0914b4dfd56aa0295dcb33d86986d1d59a81a6c99ccea"}
Nov 03 00:32:17.185829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.185791    6000 scope.go:122] "RemoveContainer" containerID="e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0"
Nov 03 00:32:17.198108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.198086    6000 scope.go:122] "RemoveContainer" containerID="e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0"
Nov 03 00:32:17.198479 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:17.198436    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0\": container with ID starting with e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0 not found: ID does not exist" containerID="e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0"
Nov 03 00:32:17.198551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.198491    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0"} err="failed to get container status \"e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0\": rpc error: code = NotFound desc = could not find container \"e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0\": container with ID starting with e1356498f4f65397c067a066e9fdf0a1bdecdc782ba48633ad430e2764ea9bb0 not found: ID does not exist"
Nov 03 00:32:17.235612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.235589    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"]
Nov 03 00:32:17.246701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:17.246678    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-2df502-6b69d7fd6f-mpzhw"]
Nov 03 00:32:18.568137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:18.568103    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de43d331-7219-4337-a7b4-dcb90588e10f" path="/var/lib/kubelet/pods/de43d331-7219-4337-a7b4-dcb90588e10f/volumes"
Nov 03 00:32:19.898903 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:19.898855    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:32:22.724075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:22.724027    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"]
Nov 03 00:32:22.724610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:22.724310    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy" containerID="cri-o://4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4" gracePeriod=30
Nov 03 00:32:24.295718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:24.295679    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:32:24.296139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:24.295748    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:32:25.145927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:25.145890    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:32:28.031330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.031304    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:32:28.205938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.205907    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-certs\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.205953    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-envoy\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.205986    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-podinfo\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206050    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-data\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206128    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-socket\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206188    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-token\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-token\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206240    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-kube-api-access-5z4dv\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-kube-api-access-5z4dv\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206350    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-credential-socket\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206426    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f097aed8-d75f-4dda-ac53-27eb137685a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f097aed8-d75f-4dda-ac53-27eb137685a5-istiod-ca-cert\") pod \"f097aed8-d75f-4dda-ac53-27eb137685a5\" (UID: \"f097aed8-d75f-4dda-ac53-27eb137685a5\") "
Nov 03 00:32:28.206797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206235    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-certs" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:28.206797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206473    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-data" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:28.206797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206517    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-socket" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:28.206797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206719    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-credential-socket" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:28.206797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.206802    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f097aed8-d75f-4dda-ac53-27eb137685a5-istiod-ca-cert" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:28.207296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.207276    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.207381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.207303    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f097aed8-d75f-4dda-ac53-27eb137685a5-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.207381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.207318    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.207381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.207331    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.207381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.207344    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.208484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.208461    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-podinfo" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:28.208575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.208499    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-envoy" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:28.208575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.208527    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-token" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:28.208644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.208619    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-kube-api-access-5z4dv" pod "f097aed8-d75f-4dda-ac53-27eb137685a5" (UID: "f097aed8-d75f-4dda-ac53-27eb137685a5"). InnerVolumeSpecName "kube-api-access-5z4dv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:28.284410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.284332    6000 generic.go:356] "Generic (PLEG): container finished" podID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerID="4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4" exitCode=0
Nov 03 00:32:28.284410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.284379    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"
Nov 03 00:32:28.284410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.284390    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" event={"ID":"f097aed8-d75f-4dda-ac53-27eb137685a5","Type":"ContainerDied","Data":"4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4"}
Nov 03 00:32:28.284630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.284434    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx" event={"ID":"f097aed8-d75f-4dda-ac53-27eb137685a5","Type":"ContainerDied","Data":"5f99a54adffddabbfb9d5c6b227480e05909c61979a2da766f81556931abc45c"}
Nov 03 00:32:28.284630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.284454    6000 scope.go:122] "RemoveContainer" containerID="4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4"
Nov 03 00:32:28.295459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.295442    6000 scope.go:122] "RemoveContainer" containerID="1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae"
Nov 03 00:32:28.305298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.305277    6000 scope.go:122] "RemoveContainer" containerID="4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4"
Nov 03 00:32:28.305579 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:28.305561    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4\": container with ID starting with 4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4 not found: ID does not exist" containerID="4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4"
Nov 03 00:32:28.305627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.305588    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4"} err="failed to get container status \"4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4\": rpc error: code = NotFound desc = could not find container \"4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4\": container with ID starting with 4b6ecf5b894bcc42479c06119e5d300b9a9e80570ece9a4ed4813bd9b1fba3b4 not found: ID does not exist"
Nov 03 00:32:28.305627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.305604    6000 scope.go:122] "RemoveContainer" containerID="1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae"
Nov 03 00:32:28.305835 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:28.305817    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae\": container with ID starting with 1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae not found: ID does not exist" containerID="1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae"
Nov 03 00:32:28.305877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.305842    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae"} err="failed to get container status \"1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae\": rpc error: code = NotFound desc = could not find container \"1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae\": container with ID starting with 1bac0d99e496a6f44f093d3034d2309bf3dc7bda265c28529c9860c059bda3ae not found: ID does not exist"
Nov 03 00:32:28.308785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.308755    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.308785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.308791    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5z4dv\" (UniqueName: \"kubernetes.io/projected/f097aed8-d75f-4dda-ac53-27eb137685a5-kube-api-access-5z4dv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.308879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.308801    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.308879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.308810    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f097aed8-d75f-4dda-ac53-27eb137685a5-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:28.333977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.333955    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"]
Nov 03 00:32:28.337634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.337613    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-2df502-openshift-default-5b67f8cbb9-qggdx"]
Nov 03 00:32:28.568372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.568302    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" path="/var/lib/kubelet/pods/f097aed8-d75f-4dda-ac53-27eb137685a5/volumes"
Nov 03 00:32:28.899787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.899750    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"]
Nov 03 00:32:28.933671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:28.933586    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"]
Nov 03 00:32:30.549741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.549704    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"]
Nov 03 00:32:30.550165 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.549919    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz" podUID="2b760644-262e-4470-b3a2-e462d3479d80" containerName="maas-api" containerID="cri-o://aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948" gracePeriod=30
Nov 03 00:32:30.831534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.831504    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:32:30.959496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.959464    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls\") pod \"2b760644-262e-4470-b3a2-e462d3479d80\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") "
Nov 03 00:32:30.959665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.959585    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b760644-262e-4470-b3a2-e462d3479d80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b760644-262e-4470-b3a2-e462d3479d80-openshift-service-ca\") pod \"2b760644-262e-4470-b3a2-e462d3479d80\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") "
Nov 03 00:32:30.959710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.959693    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-metrics-certs\") pod \"2b760644-262e-4470-b3a2-e462d3479d80\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") "
Nov 03 00:32:30.959750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.959729    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b760644-262e-4470-b3a2-e462d3479d80-kube-api-access-mz8kv\" (UniqueName: \"kubernetes.io/projected/2b760644-262e-4470-b3a2-e462d3479d80-kube-api-access-mz8kv\") pod \"2b760644-262e-4470-b3a2-e462d3479d80\" (UID: \"2b760644-262e-4470-b3a2-e462d3479d80\") "
Nov 03 00:32:30.960027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.959998    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b760644-262e-4470-b3a2-e462d3479d80-openshift-service-ca" pod "2b760644-262e-4470-b3a2-e462d3479d80" (UID: "2b760644-262e-4470-b3a2-e462d3479d80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:30.960360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.960309    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b760644-262e-4470-b3a2-e462d3479d80-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:30.961740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.961716    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls" pod "2b760644-262e-4470-b3a2-e462d3479d80" (UID: "2b760644-262e-4470-b3a2-e462d3479d80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:30.961845 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.961826    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-metrics-certs" pod "2b760644-262e-4470-b3a2-e462d3479d80" (UID: "2b760644-262e-4470-b3a2-e462d3479d80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:30.962010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:30.961993    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b760644-262e-4470-b3a2-e462d3479d80-kube-api-access-mz8kv" pod "2b760644-262e-4470-b3a2-e462d3479d80" (UID: "2b760644-262e-4470-b3a2-e462d3479d80"). InnerVolumeSpecName "kube-api-access-mz8kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:31.074334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.063465    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:31.074334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.063504    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mz8kv\" (UniqueName: \"kubernetes.io/projected/2b760644-262e-4470-b3a2-e462d3479d80-kube-api-access-mz8kv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:31.074334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.063519    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2b760644-262e-4470-b3a2-e462d3479d80-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:31.307212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.307174    6000 generic.go:356] "Generic (PLEG): container finished" podID="2b760644-262e-4470-b3a2-e462d3479d80" containerID="aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948" exitCode=0
Nov 03 00:32:31.307437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.307229    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"
Nov 03 00:32:31.307437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.307287    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz" event={"ID":"2b760644-262e-4470-b3a2-e462d3479d80","Type":"ContainerDied","Data":"aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948"}
Nov 03 00:32:31.307437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.307338    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz" event={"ID":"2b760644-262e-4470-b3a2-e462d3479d80","Type":"ContainerDied","Data":"5c53729e6c04ba15fa3a3c47ff4f1eb22cf756b6a3f1866fb3c6914d25d9c002"}
Nov 03 00:32:31.307437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.307362    6000 scope.go:122] "RemoveContainer" containerID="aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948"
Nov 03 00:32:31.320173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.319981    6000 scope.go:122] "RemoveContainer" containerID="aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948"
Nov 03 00:32:31.321102 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:31.321071    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948\": container with ID starting with aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948 not found: ID does not exist" containerID="aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948"
Nov 03 00:32:31.321228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.321110    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948"} err="failed to get container status \"aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948\": rpc error: code = NotFound desc = could not find container \"aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948\": container with ID starting with aac9e4c86250a9a9bfc616e68858b25208d0feb9f8408033bc1e549a79c2e948 not found: ID does not exist"
Nov 03 00:32:31.356108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.356088    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"]
Nov 03 00:32:31.362986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:31.362965    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-b6cd31-7c744b878-29jcz"]
Nov 03 00:32:32.569423 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:32.569395    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b760644-262e-4470-b3a2-e462d3479d80" path="/var/lib/kubelet/pods/2b760644-262e-4470-b3a2-e462d3479d80/volumes"
Nov 03 00:32:33.900104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:33.900059    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m" podUID="3f5f82ec-61ba-456b-87b9-467f85256fd3" containerName="payload-processing" containerID="cri-o://db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb" gracePeriod=25
Nov 03 00:32:34.197772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.197749    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:32:34.301286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.294323    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb" podUID="4a616e87-408c-437c-94c5-bbbb7454d614" containerName="payload-pre-processing" containerID="cri-o://44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1" gracePeriod=25
Nov 03 00:32:34.329711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.318722    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f5f82ec-61ba-456b-87b9-467f85256fd3-kube-api-access-gs5t6\" (UniqueName: \"kubernetes.io/projected/3f5f82ec-61ba-456b-87b9-467f85256fd3-kube-api-access-gs5t6\") pod \"3f5f82ec-61ba-456b-87b9-467f85256fd3\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") "
Nov 03 00:32:34.329711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.318801    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f5f82ec-61ba-456b-87b9-467f85256fd3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5f82ec-61ba-456b-87b9-467f85256fd3-plugins-config-volume\") pod \"3f5f82ec-61ba-456b-87b9-467f85256fd3\" (UID: \"3f5f82ec-61ba-456b-87b9-467f85256fd3\") "
Nov 03 00:32:34.329711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.323962    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5f82ec-61ba-456b-87b9-467f85256fd3-plugins-config-volume" pod "3f5f82ec-61ba-456b-87b9-467f85256fd3" (UID: "3f5f82ec-61ba-456b-87b9-467f85256fd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:34.419134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.416437    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5f82ec-61ba-456b-87b9-467f85256fd3-kube-api-access-gs5t6" pod "3f5f82ec-61ba-456b-87b9-467f85256fd3" (UID: "3f5f82ec-61ba-456b-87b9-467f85256fd3"). InnerVolumeSpecName "kube-api-access-gs5t6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:34.432330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.421348    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs5t6\" (UniqueName: \"kubernetes.io/projected/3f5f82ec-61ba-456b-87b9-467f85256fd3-kube-api-access-gs5t6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:34.432330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.421384    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3f5f82ec-61ba-456b-87b9-467f85256fd3-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:34.461306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.460786    6000 generic.go:356] "Generic (PLEG): container finished" podID="3f5f82ec-61ba-456b-87b9-467f85256fd3" containerID="db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb" exitCode=0
Nov 03 00:32:34.461306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.460983    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m" event={"ID":"3f5f82ec-61ba-456b-87b9-467f85256fd3","Type":"ContainerDied","Data":"db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb"}
Nov 03 00:32:34.461306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.461036    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m" event={"ID":"3f5f82ec-61ba-456b-87b9-467f85256fd3","Type":"ContainerDied","Data":"cdec8d9d361bd450683754da274cf2676c36e677b79a34c95b829e068300ea2d"}
Nov 03 00:32:34.461306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.461064    6000 scope.go:122] "RemoveContainer" containerID="db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb"
Nov 03 00:32:34.461306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.461282    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"
Nov 03 00:32:34.641383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.641317    6000 scope.go:122] "RemoveContainer" containerID="db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb"
Nov 03 00:32:34.641835 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:34.641811    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb\": container with ID starting with db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb not found: ID does not exist" containerID="db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb"
Nov 03 00:32:34.641947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.641847    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb"} err="failed to get container status \"db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb\": rpc error: code = NotFound desc = could not find container \"db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb\": container with ID starting with db4979c621bcc85a6d8c976150ebb721906f67d5ce71cb52ed9049f00b37d8bb not found: ID does not exist"
Nov 03 00:32:34.705823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.705794    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"]
Nov 03 00:32:34.705997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.705839    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-68738d-5c647d7f6f-m288m"]
Nov 03 00:32:34.768806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.768763    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:32:34.840023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.839987    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a616e87-408c-437c-94c5-bbbb7454d614-kube-api-access-wnc6g\" (UniqueName: \"kubernetes.io/projected/4a616e87-408c-437c-94c5-bbbb7454d614-kube-api-access-wnc6g\") pod \"4a616e87-408c-437c-94c5-bbbb7454d614\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") "
Nov 03 00:32:34.840240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.840226    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a616e87-408c-437c-94c5-bbbb7454d614-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a616e87-408c-437c-94c5-bbbb7454d614-plugins-config-volume\") pod \"4a616e87-408c-437c-94c5-bbbb7454d614\" (UID: \"4a616e87-408c-437c-94c5-bbbb7454d614\") "
Nov 03 00:32:34.840687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.840656    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a616e87-408c-437c-94c5-bbbb7454d614-plugins-config-volume" pod "4a616e87-408c-437c-94c5-bbbb7454d614" (UID: "4a616e87-408c-437c-94c5-bbbb7454d614"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:34.840835 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.840810    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a616e87-408c-437c-94c5-bbbb7454d614-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:34.842511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.842488    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a616e87-408c-437c-94c5-bbbb7454d614-kube-api-access-wnc6g" pod "4a616e87-408c-437c-94c5-bbbb7454d614" (UID: "4a616e87-408c-437c-94c5-bbbb7454d614"). InnerVolumeSpecName "kube-api-access-wnc6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:34.942473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:34.942402    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnc6g\" (UniqueName: \"kubernetes.io/projected/4a616e87-408c-437c-94c5-bbbb7454d614-kube-api-access-wnc6g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:35.469211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.469174    6000 generic.go:356] "Generic (PLEG): container finished" podID="4a616e87-408c-437c-94c5-bbbb7454d614" containerID="44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1" exitCode=0
Nov 03 00:32:35.469394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.469221    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"
Nov 03 00:32:35.469394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.469269    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb" event={"ID":"4a616e87-408c-437c-94c5-bbbb7454d614","Type":"ContainerDied","Data":"44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1"}
Nov 03 00:32:35.469394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.469309    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb" event={"ID":"4a616e87-408c-437c-94c5-bbbb7454d614","Type":"ContainerDied","Data":"8388cfb8d63e083d4bc0fe75bdc59866f6b56506ad1c444417e81b6f8415edda"}
Nov 03 00:32:35.469394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.469329    6000 scope.go:122] "RemoveContainer" containerID="44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1"
Nov 03 00:32:35.480599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.480577    6000 scope.go:122] "RemoveContainer" containerID="44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1"
Nov 03 00:32:35.480906 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:35.480879    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1\": container with ID starting with 44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1 not found: ID does not exist" containerID="44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1"
Nov 03 00:32:35.480972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.480917    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1"} err="failed to get container status \"44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1\": rpc error: code = NotFound desc = could not find container \"44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1\": container with ID starting with 44711ba55d892a6f78265a45186d507ce65eea189bf8d2a5f3542c216a6b67d1 not found: ID does not exist"
Nov 03 00:32:35.539709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.539683    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"]
Nov 03 00:32:35.563883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:35.563846    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-68738d-f487df4cf-s4pvb"]
Nov 03 00:32:36.618282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:36.612401    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f5f82ec-61ba-456b-87b9-467f85256fd3" path="/var/lib/kubelet/pods/3f5f82ec-61ba-456b-87b9-467f85256fd3/volumes"
Nov 03 00:32:36.618282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:36.612962    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a616e87-408c-437c-94c5-bbbb7454d614" path="/var/lib/kubelet/pods/4a616e87-408c-437c-94c5-bbbb7454d614/volumes"
Nov 03 00:32:41.999172 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999134    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"]
Nov 03 00:32:41.999825 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999810    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy"
Nov 03 00:32:41.999870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999827    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy"
Nov 03 00:32:41.999870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999837    6000 cpu_manager.go:414] "containerMap: removing container" podUID="4a616e87-408c-437c-94c5-bbbb7454d614" containerName="payload-pre-processing"
Nov 03 00:32:41.999870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999842    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a616e87-408c-437c-94c5-bbbb7454d614" containerName="payload-pre-processing"
Nov 03 00:32:41.999870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999859    6000 cpu_manager.go:414] "containerMap: removing container" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy"
Nov 03 00:32:41.999870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999864    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999877    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999886    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999895    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2b760644-262e-4470-b3a2-e462d3479d80" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999900    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b760644-262e-4470-b3a2-e462d3479d80" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999908    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999912    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999920    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999924    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999932    6000 cpu_manager.go:414] "containerMap: removing container" podUID="de43d331-7219-4337-a7b4-dcb90588e10f" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999937    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de43d331-7219-4337-a7b4-dcb90588e10f" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999942    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1f1ddfe4-135a-4efc-af63-ba784dabca6a" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999948    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f1ddfe4-135a-4efc-af63-ba784dabca6a" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999968    6000 cpu_manager.go:414] "containerMap: removing container" podUID="3f5f82ec-61ba-456b-87b9-467f85256fd3" containerName="payload-processing"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999973    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f5f82ec-61ba-456b-87b9-467f85256fd3" containerName="payload-processing"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999981    6000 cpu_manager.go:414] "containerMap: removing container" podUID="5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999985    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" containerName="maas-api"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:41.999998    6000 cpu_manager.go:414] "containerMap: removing container" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy"
Nov 03 00:32:42.000027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000003    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000092    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b760644-262e-4470-b3a2-e462d3479d80" containerName="maas-api"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000101    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c43838a-640d-4fe9-81c6-f1cb1d8eb327" containerName="istio-proxy"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000107    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000114    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="de43d331-7219-4337-a7b4-dcb90588e10f" containerName="maas-api"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000124    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f1ddfe4-135a-4efc-af63-ba784dabca6a" containerName="maas-api"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000133    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000140    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f5f82ec-61ba-456b-87b9-467f85256fd3" containerName="payload-processing"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000148    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f8a9c80-0b03-4ac6-9dab-97fb3da726fe" containerName="maas-api"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000154    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ba0b79-946a-44df-b0b3-befc1d76751f" containerName="istio-proxy"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000162    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a616e87-408c-437c-94c5-bbbb7454d614" containerName="payload-pre-processing"
Nov 03 00:32:42.000618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.000170    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="3101c60a-bbf0-41d3-99cf-4f81bff4bff8" containerName="istio-proxy"
Nov 03 00:32:42.005116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.005094    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.021958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.021933    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"]
Nov 03 00:32:42.030215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.030179    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-5d51a5\""
Nov 03 00:32:42.030215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.030196    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-5d51a5-dockercfg-dq8sd\""
Nov 03 00:32:42.060384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.060345    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"]
Nov 03 00:32:42.061006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.060989    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy"
Nov 03 00:32:42.061006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.061005    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy"
Nov 03 00:32:42.061153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.061122    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef1b4163-a397-484b-973f-a4d8fb9f6e88" containerName="istio-proxy"
Nov 03 00:32:42.061153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.061131    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f097aed8-d75f-4dda-ac53-27eb137685a5" containerName="istio-proxy"
Nov 03 00:32:42.065002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.064620    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.080205 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.080176    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"]
Nov 03 00:32:42.326307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.326190    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ea1a6-b244-4cc2-b867-5812fd2151d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.326897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.326865    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6efa864-b60d-48b0-a1a0-6194a19578eb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.327190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.327173    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzrfr\" (UniqueName: \"kubernetes.io/projected/c6efa864-b60d-48b0-a1a0-6194a19578eb-kube-api-access-vzrfr\") pod \"payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.327353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.327336    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b86t\" (UniqueName: \"kubernetes.io/projected/660ea1a6-b244-4cc2-b867-5812fd2151d2-kube-api-access-7b86t\") pod \"payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.428412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.428364    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6efa864-b60d-48b0-a1a0-6194a19578eb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.428603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.428520    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzrfr\" (UniqueName: \"kubernetes.io/projected/c6efa864-b60d-48b0-a1a0-6194a19578eb-kube-api-access-vzrfr\") pod \"payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.428603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.428555    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b86t\" (UniqueName: \"kubernetes.io/projected/660ea1a6-b244-4cc2-b867-5812fd2151d2-kube-api-access-7b86t\") pod \"payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.428722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.428644    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ea1a6-b244-4cc2-b867-5812fd2151d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.429214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.429189    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6efa864-b60d-48b0-a1a0-6194a19578eb-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.429475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.429459    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ea1a6-b244-4cc2-b867-5812fd2151d2-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.446858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.446834    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b86t\" (UniqueName: \"kubernetes.io/projected/660ea1a6-b244-4cc2-b867-5812fd2151d2-kube-api-access-7b86t\") pod \"payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.447123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.447103    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzrfr\" (UniqueName: \"kubernetes.io/projected/c6efa864-b60d-48b0-a1a0-6194a19578eb-kube-api-access-vzrfr\") pod \"payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.619983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.619955    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:42.683227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.683198    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:42.923967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:42.923938    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"]
Nov 03 00:32:42.924104 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:32:42.924067    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6efa864_b60d_48b0_a1a0_6194a19578eb.slice/crio-30ef9087117376ab196fd4a955c8d60e093855c585c0e8b6dc8e1bb29418d4a2 WatchSource:0}: Error finding container 30ef9087117376ab196fd4a955c8d60e093855c585c0e8b6dc8e1bb29418d4a2: Status 404 returned error can't find the container with id 30ef9087117376ab196fd4a955c8d60e093855c585c0e8b6dc8e1bb29418d4a2
Nov 03 00:32:43.000401 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:32:43.000360    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660ea1a6_b244_4cc2_b867_5812fd2151d2.slice/crio-57a2a485266fd89fb4250885330fc5bad9d0290c8330a7940151aad65d884221 WatchSource:0}: Error finding container 57a2a485266fd89fb4250885330fc5bad9d0290c8330a7940151aad65d884221: Status 404 returned error can't find the container with id 57a2a485266fd89fb4250885330fc5bad9d0290c8330a7940151aad65d884221
Nov 03 00:32:43.001552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.001527    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"]
Nov 03 00:32:43.635677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.635635    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx" event={"ID":"c6efa864-b60d-48b0-a1a0-6194a19578eb","Type":"ContainerStarted","Data":"ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914"}
Nov 03 00:32:43.635677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.635685    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx" event={"ID":"c6efa864-b60d-48b0-a1a0-6194a19578eb","Type":"ContainerStarted","Data":"30ef9087117376ab196fd4a955c8d60e093855c585c0e8b6dc8e1bb29418d4a2"}
Nov 03 00:32:43.636542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.636524    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:43.638598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.638567    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89" event={"ID":"660ea1a6-b244-4cc2-b867-5812fd2151d2","Type":"ContainerStarted","Data":"345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44"}
Nov 03 00:32:43.638804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.638621    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89" event={"ID":"660ea1a6-b244-4cc2-b867-5812fd2151d2","Type":"ContainerStarted","Data":"57a2a485266fd89fb4250885330fc5bad9d0290c8330a7940151aad65d884221"}
Nov 03 00:32:43.639196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.639174    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:43.674572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.674524    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx" podStartSLOduration=2.674510026 podStartE2EDuration="2.674510026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:43.672370213 +0000 UTC m=+1601.227722238" watchObservedRunningTime="2026-11-03 00:32:43.674510026 +0000 UTC m=+1601.229862050"
Nov 03 00:32:43.711457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:43.711405    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89" podStartSLOduration=1.711391936 podStartE2EDuration="1.711391936s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:32:43.70938866 +0000 UTC m=+1601.264740680" watchObservedRunningTime="2026-11-03 00:32:43.711391936 +0000 UTC m=+1601.266743959"
Nov 03 00:32:48.267098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.267060    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"]
Nov 03 00:32:48.267667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.267319    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz" podUID="cd3dc0fe-bd42-43fa-976f-404b5dad83db" containerName="maas-api" containerID="cri-o://3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126" gracePeriod=30
Nov 03 00:32:48.655281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.633652    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:32:48.740876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.738797    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-maas-api-tls\") pod \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") "
Nov 03 00:32:48.740876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.738923    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd3dc0fe-bd42-43fa-976f-404b5dad83db-kube-api-access-v7z86\" (UniqueName: \"kubernetes.io/projected/cd3dc0fe-bd42-43fa-976f-404b5dad83db-kube-api-access-v7z86\") pod \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") "
Nov 03 00:32:48.740876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.739056    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-metrics-certs\") pod \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") "
Nov 03 00:32:48.740876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.739114    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd3dc0fe-bd42-43fa-976f-404b5dad83db-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3dc0fe-bd42-43fa-976f-404b5dad83db-openshift-service-ca\") pod \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\" (UID: \"cd3dc0fe-bd42-43fa-976f-404b5dad83db\") "
Nov 03 00:32:48.740876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.740211    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3dc0fe-bd42-43fa-976f-404b5dad83db-openshift-service-ca" pod "cd3dc0fe-bd42-43fa-976f-404b5dad83db" (UID: "cd3dc0fe-bd42-43fa-976f-404b5dad83db"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:48.750026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.749984    6000 generic.go:356] "Generic (PLEG): container finished" podID="cd3dc0fe-bd42-43fa-976f-404b5dad83db" containerID="3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126" exitCode=0
Nov 03 00:32:48.750492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.750063    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz" event={"ID":"cd3dc0fe-bd42-43fa-976f-404b5dad83db","Type":"ContainerDied","Data":"3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126"}
Nov 03 00:32:48.750492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.750111    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz" event={"ID":"cd3dc0fe-bd42-43fa-976f-404b5dad83db","Type":"ContainerDied","Data":"7745b7a0b67011deaa337db14c4769503381a719284d7d9ae4c7edf264d20091"}
Nov 03 00:32:48.750492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.750139    6000 scope.go:122] "RemoveContainer" containerID="3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126"
Nov 03 00:32:48.750492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.750316    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"
Nov 03 00:32:48.798641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.798589    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-maas-api-tls" pod "cd3dc0fe-bd42-43fa-976f-404b5dad83db" (UID: "cd3dc0fe-bd42-43fa-976f-404b5dad83db"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:48.816594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.816205    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd3dc0fe-bd42-43fa-976f-404b5dad83db-kube-api-access-v7z86" pod "cd3dc0fe-bd42-43fa-976f-404b5dad83db" (UID: "cd3dc0fe-bd42-43fa-976f-404b5dad83db"). InnerVolumeSpecName "kube-api-access-v7z86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:48.828953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.828888    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-metrics-certs" pod "cd3dc0fe-bd42-43fa-976f-404b5dad83db" (UID: "cd3dc0fe-bd42-43fa-976f-404b5dad83db"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:32:48.871661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.847607    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:48.871661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.847651    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7z86\" (UniqueName: \"kubernetes.io/projected/cd3dc0fe-bd42-43fa-976f-404b5dad83db-kube-api-access-v7z86\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:48.871661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.847679    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cd3dc0fe-bd42-43fa-976f-404b5dad83db-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:48.871661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.847699    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3dc0fe-bd42-43fa-976f-404b5dad83db-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:48.907468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.907439    6000 scope.go:122] "RemoveContainer" containerID="3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126"
Nov 03 00:32:48.909739 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:48.909622    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126\": container with ID starting with 3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126 not found: ID does not exist" containerID="3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126"
Nov 03 00:32:48.909739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:48.909668    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126"} err="failed to get container status \"3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126\": rpc error: code = NotFound desc = could not find container \"3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126\": container with ID starting with 3a41f2174a053ba5d0ac375934dabf31e4acd248e908a2886fb050ecdd4a4126 not found: ID does not exist"
Nov 03 00:32:49.095844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:49.095809    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"]
Nov 03 00:32:49.100641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:49.100615    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-b-68738d-cf474d67-q6chz"]
Nov 03 00:32:49.900542 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:49.900503    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:32:50.456212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:50.456086    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"]
Nov 03 00:32:50.456720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:50.456469    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy" containerID="cri-o://e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb" gracePeriod=30
Nov 03 00:32:50.567990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:50.567957    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd3dc0fe-bd42-43fa-976f-404b5dad83db" path="/var/lib/kubelet/pods/cd3dc0fe-bd42-43fa-976f-404b5dad83db/volumes"
Nov 03 00:32:54.078633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:54.078596    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"]
Nov 03 00:32:54.687515 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:54.687461    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:32:55.662903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.662870    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:32:55.787557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.787528    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:32:55.839102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.839072    6000 generic.go:356] "Generic (PLEG): container finished" podID="d42bd665-182b-4e8d-870f-3793b098957d" containerID="e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb" exitCode=0
Nov 03 00:32:55.839347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.839138    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"
Nov 03 00:32:55.839347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.839157    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" event={"ID":"d42bd665-182b-4e8d-870f-3793b098957d","Type":"ContainerDied","Data":"e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb"}
Nov 03 00:32:55.839347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.839223    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw" event={"ID":"d42bd665-182b-4e8d-870f-3793b098957d","Type":"ContainerDied","Data":"9202ff350e44c5cc5baa8d9fb2009f00a85bffbc8e50e796803d27f88aecbba3"}
Nov 03 00:32:55.839347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.839270    6000 scope.go:122] "RemoveContainer" containerID="e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb"
Nov 03 00:32:55.853717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.853696    6000 scope.go:122] "RemoveContainer" containerID="e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb"
Nov 03 00:32:55.854765 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:32:55.854736    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb\": container with ID starting with e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb not found: ID does not exist" containerID="e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb"
Nov 03 00:32:55.854943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.854911    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb"} err="failed to get container status \"e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb\": rpc error: code = NotFound desc = could not find container \"e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb\": container with ID starting with e95a3b76f0735c1bb4c4cfd24c5c3252c89d68c90aa1efd39a62659dccf497fb not found: ID does not exist"
Nov 03 00:32:55.901384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901355    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-credential-socket\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901502    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d42bd665-182b-4e8d-870f-3793b098957d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42bd665-182b-4e8d-870f-3793b098957d-istiod-ca-cert\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901551    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-istio-token\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-istio-token\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901631    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d42bd665-182b-4e8d-870f-3793b098957d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42bd665-182b-4e8d-870f-3793b098957d-istio-podinfo\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901639    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-credential-socket" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.901681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901656    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-certs\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901675    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-socket\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901694    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-kube-api-access-ckz5g\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-kube-api-access-ckz5g\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901740    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-envoy\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.901871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901864    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-data\") pod \"d42bd665-182b-4e8d-870f-3793b098957d\" (UID: \"d42bd665-182b-4e8d-870f-3793b098957d\") "
Nov 03 00:32:55.902064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902017    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-certs" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.902140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.901917    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d42bd665-182b-4e8d-870f-3793b098957d-istiod-ca-cert" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:32:55.902140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902098    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-socket" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.902330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902307    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-data" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.902828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902804    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.902940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902833    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.902940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902850    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d42bd665-182b-4e8d-870f-3793b098957d-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.902940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902865    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.902940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.902879    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:55.904179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.904153    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-istio-token" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:55.904308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.904178    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-envoy" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:32:55.904375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.904309    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-kube-api-access-ckz5g" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "kube-api-access-ckz5g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:32:55.904375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:55.904344    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d42bd665-182b-4e8d-870f-3793b098957d-istio-podinfo" pod "d42bd665-182b-4e8d-870f-3793b098957d" (UID: "d42bd665-182b-4e8d-870f-3793b098957d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:32:56.004631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:56.004548    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.004631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:56.004579    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d42bd665-182b-4e8d-870f-3793b098957d-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.004631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:56.004589    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckz5g\" (UniqueName: \"kubernetes.io/projected/d42bd665-182b-4e8d-870f-3793b098957d-kube-api-access-ckz5g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.004631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:56.004601    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d42bd665-182b-4e8d-870f-3793b098957d-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:32:56.184306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:56.184236    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"]
Nov 03 00:32:56.195044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:56.195017    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-c4ckw"]
Nov 03 00:32:56.568269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:56.568220    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d42bd665-182b-4e8d-870f-3793b098957d" path="/var/lib/kubelet/pods/d42bd665-182b-4e8d-870f-3793b098957d/volumes"
Nov 03 00:32:58.898204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.898154    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"]
Nov 03 00:32:58.898995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.898955    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy"
Nov 03 00:32:58.898995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.898985    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy"
Nov 03 00:32:58.899159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.899011    6000 cpu_manager.go:414] "containerMap: removing container" podUID="cd3dc0fe-bd42-43fa-976f-404b5dad83db" containerName="maas-api"
Nov 03 00:32:58.899159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.899016    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd3dc0fe-bd42-43fa-976f-404b5dad83db" containerName="maas-api"
Nov 03 00:32:58.899159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.899116    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d42bd665-182b-4e8d-870f-3793b098957d" containerName="istio-proxy"
Nov 03 00:32:58.899159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.899128    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd3dc0fe-bd42-43fa-976f-404b5dad83db" containerName="maas-api"
Nov 03 00:32:58.904263 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.904231    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:58.922596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.922568    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"]
Nov 03 00:32:58.930546 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:58.930517    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-praxis-a-5d51a5-serving-cert\""
Nov 03 00:32:59.048489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.048452    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.048691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.048671    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.048752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.048707    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.048752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.048727    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm9m2\" (UniqueName: \"kubernetes.io/projected/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-kube-api-access-lm9m2\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.079626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.079578    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" podUID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" containerName="payload-pre-processing" containerID="cri-o://4bbab479da1f99417c6b4bf313fad2bc72988f599c5bfbea092cada2044c264d" gracePeriod=25
Nov 03 00:32:59.153159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.153078    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.153159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.153127    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.153159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.153148    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm9m2\" (UniqueName: \"kubernetes.io/projected/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-kube-api-access-lm9m2\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.153500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.153243    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.155243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.155214    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-openshift-service-ca\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.159306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.157966    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-maas-api-tls\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.159972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.159945    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-metrics-certs\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.172984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.172960    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm9m2\" (UniqueName: \"kubernetes.io/projected/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-kube-api-access-lm9m2\") pod \"maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.218919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.218881    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:32:59.517863 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:32:59.517818    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b151cdb_1789_40cb_bc0c_c6d7f75eef2d.slice/crio-0b8224407aca690c627ee677a2a6915569df942ca3ed0e4e4e42dffd5bbdf278 WatchSource:0}: Error finding container 0b8224407aca690c627ee677a2a6915569df942ca3ed0e4e4e42dffd5bbdf278: Status 404 returned error can't find the container with id 0b8224407aca690c627ee677a2a6915569df942ca3ed0e4e4e42dffd5bbdf278
Nov 03 00:32:59.518351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.518330    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"]
Nov 03 00:32:59.519466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.519447    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:32:59.884316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:32:59.884029    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh" event={"ID":"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d","Type":"ContainerStarted","Data":"0b8224407aca690c627ee677a2a6915569df942ca3ed0e4e4e42dffd5bbdf278"}
Nov 03 00:33:00.934298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:00.933819    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh" event={"ID":"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d","Type":"ContainerStarted","Data":"1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206"}
Nov 03 00:33:00.934298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:00.934022    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:33:00.975331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:00.975231    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh" podStartSLOduration=2.560099211 podStartE2EDuration="2.975212381s" totalImagesPullingTime="415.11317ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:32:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:00.974488944 +0000 UTC m=+1618.529840968" watchObservedRunningTime="2026-11-03 00:33:00.975212381 +0000 UTC m=+1618.530564414"
Nov 03 00:33:03.491025 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:03.490992    6000 scope.go:122] "RemoveContainer" containerID="a1346a3c27899c9a7f87d6856047071ed931108832eb7b84ffd42f0dec01603d"
Nov 03 00:33:04.050455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:04.050417    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:33:05.387159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.387119    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" podUID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:33:05.582595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.582555    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"]
Nov 03 00:33:05.590401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.590367    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.605870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.605839    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"]
Nov 03 00:33:05.611098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.611071    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.646946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.646852    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"]
Nov 03 00:33:05.651248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.651219    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"]
Nov 03 00:33:05.658297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.658233    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.662246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.662219    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"]
Nov 03 00:33:05.687244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.687214    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.687586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.687562    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.687834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.687814    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.687992 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.687975    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.688141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.688127    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdppb\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-kube-api-access-pdppb\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.688347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.688330    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.688525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.688509    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.688664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.688650    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.688820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.688804    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60317f80-10cb-4053-bfec-17853dc9fada-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.688981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.688966    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.689139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.689122    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.689303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.689286    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60317f80-10cb-4053-bfec-17853dc9fada-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.689454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.689437    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.689685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.689581    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.689863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.689845    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxhq4\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-kube-api-access-vxhq4\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.690078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.690061    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c572c65-36af-43d5-98e1-7cf57034a71f-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.695766 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.695738    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.695953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.695935    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.696133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.696117    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.696331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.696312    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpr9h\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-kube-api-access-hpr9h\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.696499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.696483    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.696653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.696636    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.696821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.696806    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.696977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.696962    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.697130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.697115    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.697303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.697285    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.697461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.697446    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.751619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.751581    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"]
Nov 03 00:33:05.799138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799095    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.799138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799132    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.799430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799157    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxhq4\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-kube-api-access-vxhq4\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.799430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799416    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c572c65-36af-43d5-98e1-7cf57034a71f-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.799558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799476    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.799558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799523    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.799676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799566    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.799676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799592    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpr9h\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-kube-api-access-hpr9h\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.799676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799613    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.799676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799625    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.799676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799651    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.799938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799698    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.799938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799755    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.799938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799783    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.799938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799813    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.799938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799836    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.799938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799876    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.799938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.799928    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800059    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800088    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800127    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdppb\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-kube-api-access-pdppb\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800165    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800207    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800237    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c572c65-36af-43d5-98e1-7cf57034a71f-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800238    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.800358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800338    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60317f80-10cb-4053-bfec-17853dc9fada-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.800794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800398    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.800794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800417    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.800794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800435    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.800794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800476    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60317f80-10cb-4053-bfec-17853dc9fada-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.800794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800498    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.800794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800703    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.801129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.800839    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.801129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.801062    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.801682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.801465    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.801995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.801967    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.802116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.802089    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.802300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.802248    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.802404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.802381    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.802784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.802728    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.803009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.802987    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.806393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.803468    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.806511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.803899    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.806607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.804184    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60317f80-10cb-4053-bfec-17853dc9fada-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.806713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.805019    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.806819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.805973    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60317f80-10cb-4053-bfec-17853dc9fada-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.807936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.807891    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.808662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.808612    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.819008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.818959    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxhq4\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-kube-api-access-vxhq4\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.824296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.824025    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.824296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.824063    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.824296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.824287    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdppb\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-kube-api-access-pdppb\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:05.824548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.824463    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.826069 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.826044    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpr9h\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-kube-api-access-hpr9h\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.917044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.916391    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:05.929234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.928904    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:05.984047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:05.983821    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:06.252180 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:33:06.248696    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45aef524_0f0d_4db9_aece_46db1d4e6e2f.slice/crio-bf2d8f525b4ebc7c819b2c4aaf2ff30ba9d38083d2af850a01c5161bfa1fcd07 WatchSource:0}: Error finding container bf2d8f525b4ebc7c819b2c4aaf2ff30ba9d38083d2af850a01c5161bfa1fcd07: Status 404 returned error can't find the container with id bf2d8f525b4ebc7c819b2c4aaf2ff30ba9d38083d2af850a01c5161bfa1fcd07
Nov 03 00:33:06.252180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.250323    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"]
Nov 03 00:33:06.255925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.255886    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.256021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.255974    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.256058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.256021    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.281213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.280606    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"]
Nov 03 00:33:06.282400 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:33:06.282345    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c572c65_36af_43d5_98e1_7cf57034a71f.slice/crio-e14a337b9532cf663e99af29624c07eaac4ef71f57de90e6373e7d7d8de31741 WatchSource:0}: Error finding container e14a337b9532cf663e99af29624c07eaac4ef71f57de90e6373e7d7d8de31741: Status 404 returned error can't find the container with id e14a337b9532cf663e99af29624c07eaac4ef71f57de90e6373e7d7d8de31741
Nov 03 00:33:06.285045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.285005    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.285161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.285077    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.285161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.285106    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.360277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.360033    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"]
Nov 03 00:33:06.360277 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:33:06.360182    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60317f80_10cb_4053_bfec_17853dc9fada.slice/crio-9a99e23f4e19a30b9a43f4b31d09f6b58b80b5f9fea818c1391504ff2ac9ce96 WatchSource:0}: Error finding container 9a99e23f4e19a30b9a43f4b31d09f6b58b80b5f9fea818c1391504ff2ac9ce96: Status 404 returned error can't find the container with id 9a99e23f4e19a30b9a43f4b31d09f6b58b80b5f9fea818c1391504ff2ac9ce96
Nov 03 00:33:06.364372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.364339    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.364555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.364538    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.364683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.364667    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:06.947467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:06.947436    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:33:07.006092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.006052    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" event={"ID":"60317f80-10cb-4053-bfec-17853dc9fada","Type":"ContainerStarted","Data":"ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e"}
Nov 03 00:33:07.006329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.006119    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" event={"ID":"60317f80-10cb-4053-bfec-17853dc9fada","Type":"ContainerStarted","Data":"9a99e23f4e19a30b9a43f4b31d09f6b58b80b5f9fea818c1391504ff2ac9ce96"}
Nov 03 00:33:07.008407 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.008313    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" event={"ID":"7c572c65-36af-43d5-98e1-7cf57034a71f","Type":"ContainerStarted","Data":"154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f"}
Nov 03 00:33:07.008407 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.008371    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" event={"ID":"7c572c65-36af-43d5-98e1-7cf57034a71f","Type":"ContainerStarted","Data":"e14a337b9532cf663e99af29624c07eaac4ef71f57de90e6373e7d7d8de31741"}
Nov 03 00:33:07.011241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.011217    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" event={"ID":"45aef524-0f0d-4db9-aece-46db1d4e6e2f","Type":"ContainerStarted","Data":"44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693"}
Nov 03 00:33:07.011394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.011284    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" event={"ID":"45aef524-0f0d-4db9-aece-46db1d4e6e2f","Type":"ContainerStarted","Data":"bf2d8f525b4ebc7c819b2c4aaf2ff30ba9d38083d2af850a01c5161bfa1fcd07"}
Nov 03 00:33:07.049608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.049557    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podStartSLOduration=2.049541357 podStartE2EDuration="2.049541357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:07.046580966 +0000 UTC m=+1624.601932991" watchObservedRunningTime="2026-11-03 00:33:07.049541357 +0000 UTC m=+1624.604893381"
Nov 03 00:33:07.095177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.095126    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podStartSLOduration=2.095111955 podStartE2EDuration="2.095111955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:07.093087421 +0000 UTC m=+1624.648439441" watchObservedRunningTime="2026-11-03 00:33:07.095111955 +0000 UTC m=+1624.650463978"
Nov 03 00:33:07.143733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.143660    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podStartSLOduration=2.14364211 podStartE2EDuration="2.14364211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:07.139361903 +0000 UTC m=+1624.694713939" watchObservedRunningTime="2026-11-03 00:33:07.14364211 +0000 UTC m=+1624.698994134"
Nov 03 00:33:07.928922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.928875    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:07.938280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.929454    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:07.938280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.934536    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:33:07.938280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.934591    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 00:33:07.938280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.934922    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:33:07.938280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.934955    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 00:33:07.994286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.984929    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:07.994286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.986357    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:33:07.994286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:07.986409    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 00:33:08.927714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:08.927673    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:33:08.928030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:08.928001    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 00:33:08.932687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:08.932652    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:33:08.932925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:08.932901    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 00:33:08.984861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:08.984826    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:33:08.985037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:08.984878    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 00:33:09.919248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:09.918746    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:33:09.919248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:09.918821    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 00:33:09.932794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:09.932678    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:33:09.932794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:09.932737    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 00:33:09.984923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:09.984887    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:33:09.985101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:09.984939    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 00:33:10.956331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:10.948040    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:33:10.971402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:10.971355    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 00:33:10.971688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:10.949779    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:33:10.971857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:10.971838    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 00:33:10.984286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:10.984240    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:33:10.984424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:10.984317    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 00:33:11.928281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:11.920655    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:33:11.928281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:11.920728    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 00:33:11.933278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:11.929593    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:33:11.933278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:11.929670    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 00:33:11.984773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:11.984700    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:33:11.984773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:11.984750    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 00:33:12.919709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:12.919668    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:33:12.920010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:12.919980    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 00:33:12.940733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:12.940689    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:33:12.941585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:12.941553    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 00:33:12.968322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:12.968288    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"]
Nov 03 00:33:12.985092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:12.985055    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:33:12.985753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:12.985721    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 00:33:13.180348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.180207    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252" podUID="67b3b3d1-a353-4a58-aadb-0e845ed8e755" containerName="maas-api" containerID="cri-o://3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9" gracePeriod=30
Nov 03 00:33:13.900426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.900392    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:33:13.917642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.917604    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body=
Nov 03 00:33:13.917820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.917666    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused"
Nov 03 00:33:13.930125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.930089    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 03 00:33:13.930369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.930147    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 03 00:33:13.959222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.959173    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b3b3d1-a353-4a58-aadb-0e845ed8e755-kube-api-access-8xvcx\" (UniqueName: \"kubernetes.io/projected/67b3b3d1-a353-4a58-aadb-0e845ed8e755-kube-api-access-8xvcx\") pod \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") "
Nov 03 00:33:13.959552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.959532    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b3b3d1-a353-4a58-aadb-0e845ed8e755-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b3b3d1-a353-4a58-aadb-0e845ed8e755-openshift-service-ca\") pod \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") "
Nov 03 00:33:13.959794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.959774    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-maas-api-tls\") pod \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") "
Nov 03 00:33:13.960029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.960008    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-metrics-certs\") pod \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\" (UID: \"67b3b3d1-a353-4a58-aadb-0e845ed8e755\") "
Nov 03 00:33:13.960218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.960156    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b3b3d1-a353-4a58-aadb-0e845ed8e755-openshift-service-ca" pod "67b3b3d1-a353-4a58-aadb-0e845ed8e755" (UID: "67b3b3d1-a353-4a58-aadb-0e845ed8e755"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:13.961034 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.961005    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/67b3b3d1-a353-4a58-aadb-0e845ed8e755-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:13.964742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.964699    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-metrics-certs" pod "67b3b3d1-a353-4a58-aadb-0e845ed8e755" (UID: "67b3b3d1-a353-4a58-aadb-0e845ed8e755"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:13.965435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.965408    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-maas-api-tls" pod "67b3b3d1-a353-4a58-aadb-0e845ed8e755" (UID: "67b3b3d1-a353-4a58-aadb-0e845ed8e755"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:33:13.965542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:13.965526    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b3b3d1-a353-4a58-aadb-0e845ed8e755-kube-api-access-8xvcx" pod "67b3b3d1-a353-4a58-aadb-0e845ed8e755" (UID: "67b3b3d1-a353-4a58-aadb-0e845ed8e755"). InnerVolumeSpecName "kube-api-access-8xvcx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:14.001419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.000895    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body=
Nov 03 00:33:14.001419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.000988    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused"
Nov 03 00:33:14.068035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.067967    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:14.068035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.068034    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67b3b3d1-a353-4a58-aadb-0e845ed8e755-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:14.068358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.068050    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvcx\" (UniqueName: \"kubernetes.io/projected/67b3b3d1-a353-4a58-aadb-0e845ed8e755-kube-api-access-8xvcx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:14.189768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.189726    6000 generic.go:356] "Generic (PLEG): container finished" podID="67b3b3d1-a353-4a58-aadb-0e845ed8e755" containerID="3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9" exitCode=0
Nov 03 00:33:14.189974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.189771    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"
Nov 03 00:33:14.189974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.189820    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252" event={"ID":"67b3b3d1-a353-4a58-aadb-0e845ed8e755","Type":"ContainerDied","Data":"3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9"}
Nov 03 00:33:14.189974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.189929    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252" event={"ID":"67b3b3d1-a353-4a58-aadb-0e845ed8e755","Type":"ContainerDied","Data":"ca82481e2b0c7a085a715e23580a7ec47f1c53a3dee3850ac881cf0e1c95eb66"}
Nov 03 00:33:14.190178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.189976    6000 scope.go:122] "RemoveContainer" containerID="3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9"
Nov 03 00:33:14.203287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.203232    6000 scope.go:122] "RemoveContainer" containerID="3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9"
Nov 03 00:33:14.203705 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:33:14.203687    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9\": container with ID starting with 3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9 not found: ID does not exist" containerID="3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9"
Nov 03 00:33:14.203797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.203724    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9"} err="failed to get container status \"3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9\": rpc error: code = NotFound desc = could not find container \"3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9\": container with ID starting with 3b73711f1a887971b0eedb89c6bbb3195593c11088b9bb705a07e2d40c089da9 not found: ID does not exist"
Nov 03 00:33:14.243789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.243696    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"]
Nov 03 00:33:14.248512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.248482    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-4ae722-78d9846c8c-4p252"]
Nov 03 00:33:14.572117 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:14.572031    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b3b3d1-a353-4a58-aadb-0e845ed8e755" path="/var/lib/kubelet/pods/67b3b3d1-a353-4a58-aadb-0e845ed8e755/volumes"
Nov 03 00:33:15.454803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.454753    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" podUID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 00:33:15.772070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.771955    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:15.773729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.772445    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:15.774931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.774830    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:33:15.829005 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.828968    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:15.829483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.829462    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:15.830391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.830370    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:33:15.849154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.849116    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:15.854285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.849530    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:15.854285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:15.850332    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:33:19.230032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.229994    6000 generic.go:356] "Generic (PLEG): container finished" podID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" containerID="4bbab479da1f99417c6b4bf313fad2bc72988f599c5bfbea092cada2044c264d" exitCode=0
Nov 03 00:33:19.230595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.230054    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" event={"ID":"65cf5dee-fcb9-4598-ada6-ff8209a638a8","Type":"ContainerDied","Data":"4bbab479da1f99417c6b4bf313fad2bc72988f599c5bfbea092cada2044c264d"}
Nov 03 00:33:19.368316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.368290    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:33:19.481535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.481498    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65cf5dee-fcb9-4598-ada6-ff8209a638a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cf5dee-fcb9-4598-ada6-ff8209a638a8-plugins-config-volume\") pod \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") "
Nov 03 00:33:19.481726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.481693    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65cf5dee-fcb9-4598-ada6-ff8209a638a8-kube-api-access-fr5dw\" (UniqueName: \"kubernetes.io/projected/65cf5dee-fcb9-4598-ada6-ff8209a638a8-kube-api-access-fr5dw\") pod \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\" (UID: \"65cf5dee-fcb9-4598-ada6-ff8209a638a8\") "
Nov 03 00:33:19.482197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.482172    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65cf5dee-fcb9-4598-ada6-ff8209a638a8-plugins-config-volume" pod "65cf5dee-fcb9-4598-ada6-ff8209a638a8" (UID: "65cf5dee-fcb9-4598-ada6-ff8209a638a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:19.485665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.485625    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65cf5dee-fcb9-4598-ada6-ff8209a638a8-kube-api-access-fr5dw" pod "65cf5dee-fcb9-4598-ada6-ff8209a638a8" (UID: "65cf5dee-fcb9-4598-ada6-ff8209a638a8"). InnerVolumeSpecName "kube-api-access-fr5dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:19.583376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.583347    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr5dw\" (UniqueName: \"kubernetes.io/projected/65cf5dee-fcb9-4598-ada6-ff8209a638a8-kube-api-access-fr5dw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:19.583376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:19.583372    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65cf5dee-fcb9-4598-ada6-ff8209a638a8-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:19.898172 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:33:19.898136    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:33:20.237969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:20.237893    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"
Nov 03 00:33:20.237969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:20.237931    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5" event={"ID":"65cf5dee-fcb9-4598-ada6-ff8209a638a8","Type":"ContainerDied","Data":"38b3c9d5648ed7d471e561cf21abc62639a9fb62ebc078ad04edb3757dca2634"}
Nov 03 00:33:20.238506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:20.237989    6000 scope.go:122] "RemoveContainer" containerID="4bbab479da1f99417c6b4bf313fad2bc72988f599c5bfbea092cada2044c264d"
Nov 03 00:33:20.290994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:20.290963    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"]
Nov 03 00:33:20.302425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:20.302401    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-4ae722-55765b8dcf-z9dr5"]
Nov 03 00:33:20.568482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:20.568403    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" path="/var/lib/kubelet/pods/65cf5dee-fcb9-4598-ada6-ff8209a638a8/volumes"
Nov 03 00:33:26.317464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:26.317428    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"]
Nov 03 00:33:31.319287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:31.318338    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x" podUID="d71acda9-cc37-46c4-8363-6247e99cd612" containerName="payload-pre-processing" containerID="cri-o://f9ec3259441092d297dea7060c9211a5dc0c6efbae1ffbc98555550cf09125ee" gracePeriod=25
Nov 03 00:33:31.768281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:31.767352    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:31.768281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:31.767768    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 00:33:31.882424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:31.882371    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:32.283235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.272070    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:32.359335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.341362    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:33:32.603345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.598436    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:32.691985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.690419    6000 generic.go:356] "Generic (PLEG): container finished" podID="d71acda9-cc37-46c4-8363-6247e99cd612" containerID="f9ec3259441092d297dea7060c9211a5dc0c6efbae1ffbc98555550cf09125ee" exitCode=0
Nov 03 00:33:32.719395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.719352    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:32.773280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.772553    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:32.773280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.772903    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 00:33:32.823198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.820813    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:33:32.823198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.820878    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x" event={"ID":"d71acda9-cc37-46c4-8363-6247e99cd612","Type":"ContainerDied","Data":"f9ec3259441092d297dea7060c9211a5dc0c6efbae1ffbc98555550cf09125ee"}
Nov 03 00:33:32.823198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.820967    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:33:32.823914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.823895    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:33:32.908347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.907831    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d71acda9-cc37-46c4-8363-6247e99cd612-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71acda9-cc37-46c4-8363-6247e99cd612-plugins-config-volume\") pod \"d71acda9-cc37-46c4-8363-6247e99cd612\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") "
Nov 03 00:33:32.908347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.907881    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d71acda9-cc37-46c4-8363-6247e99cd612-kube-api-access-575k4\" (UniqueName: \"kubernetes.io/projected/d71acda9-cc37-46c4-8363-6247e99cd612-kube-api-access-575k4\") pod \"d71acda9-cc37-46c4-8363-6247e99cd612\" (UID: \"d71acda9-cc37-46c4-8363-6247e99cd612\") "
Nov 03 00:33:32.971052 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.969811    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71acda9-cc37-46c4-8363-6247e99cd612-plugins-config-volume" pod "d71acda9-cc37-46c4-8363-6247e99cd612" (UID: "d71acda9-cc37-46c4-8363-6247e99cd612"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:33:32.973009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:32.972981    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71acda9-cc37-46c4-8363-6247e99cd612-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:33.140321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.140277    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71acda9-cc37-46c4-8363-6247e99cd612-kube-api-access-575k4" pod "d71acda9-cc37-46c4-8363-6247e99cd612" (UID: "d71acda9-cc37-46c4-8363-6247e99cd612"). InnerVolumeSpecName "kube-api-access-575k4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:33:33.257609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.256757    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-575k4\" (UniqueName: \"kubernetes.io/projected/d71acda9-cc37-46c4-8363-6247e99cd612-kube-api-access-575k4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:33:33.278403 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.278371    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:33:33.281775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.281749    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:33:33.711383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.711357    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"
Nov 03 00:33:33.711383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.711365    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x" event={"ID":"d71acda9-cc37-46c4-8363-6247e99cd612","Type":"ContainerDied","Data":"7835ec379ef82223ce6224a3eb02baff9deb0ed07d7c611fd3a60127c0fbcdb1"}
Nov 03 00:33:33.711893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.711438    6000 scope.go:122] "RemoveContainer" containerID="f9ec3259441092d297dea7060c9211a5dc0c6efbae1ffbc98555550cf09125ee"
Nov 03 00:33:33.762648 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.762618    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"]
Nov 03 00:33:33.768270 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:33.768227    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-191362-6cb8ff9c59-mn67x"]
Nov 03 00:33:34.014680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.014588    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"]
Nov 03 00:33:34.015360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015342    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d71acda9-cc37-46c4-8363-6247e99cd612" containerName="payload-pre-processing"
Nov 03 00:33:34.015433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015361    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d71acda9-cc37-46c4-8363-6247e99cd612" containerName="payload-pre-processing"
Nov 03 00:33:34.015433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015377    6000 cpu_manager.go:414] "containerMap: removing container" podUID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" containerName="payload-pre-processing"
Nov 03 00:33:34.015433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015382    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" containerName="payload-pre-processing"
Nov 03 00:33:34.015433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015389    6000 cpu_manager.go:414] "containerMap: removing container" podUID="67b3b3d1-a353-4a58-aadb-0e845ed8e755" containerName="maas-api"
Nov 03 00:33:34.015433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015394    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b3b3d1-a353-4a58-aadb-0e845ed8e755" containerName="maas-api"
Nov 03 00:33:34.015596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015508    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d71acda9-cc37-46c4-8363-6247e99cd612" containerName="payload-pre-processing"
Nov 03 00:33:34.015596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015519    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="65cf5dee-fcb9-4598-ada6-ff8209a638a8" containerName="payload-pre-processing"
Nov 03 00:33:34.015596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.015524    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b3b3d1-a353-4a58-aadb-0e845ed8e755" containerName="maas-api"
Nov 03 00:33:34.020481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.020459    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.037697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.037664    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-a19e67\""
Nov 03 00:33:34.039941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.039905    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"]
Nov 03 00:33:34.084932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.084905    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"]
Nov 03 00:33:34.089162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.089133    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.103220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.103196    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-a19e67-dockercfg-98dl6\""
Nov 03 00:33:34.106167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.106128    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"]
Nov 03 00:33:34.207332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.200458    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d04140-4088-45ce-818b-8c88a90d3348-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.207332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.200615    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbzn2\" (UniqueName: \"kubernetes.io/projected/56d04140-4088-45ce-818b-8c88a90d3348-kube-api-access-qbzn2\") pod \"payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.321343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.317342    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d04140-4088-45ce-818b-8c88a90d3348-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.321343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.317461    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6c415e-bcec-47aa-9ec4-28618957abe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.321343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.317559    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbzn2\" (UniqueName: \"kubernetes.io/projected/56d04140-4088-45ce-818b-8c88a90d3348-kube-api-access-qbzn2\") pod \"payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.321343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.317751    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xnhp\" (UniqueName: \"kubernetes.io/projected/7d6c415e-bcec-47aa-9ec4-28618957abe0-kube-api-access-8xnhp\") pod \"payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.321343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.318764    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d04140-4088-45ce-818b-8c88a90d3348-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.366593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.366562    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbzn2\" (UniqueName: \"kubernetes.io/projected/56d04140-4088-45ce-818b-8c88a90d3348-kube-api-access-qbzn2\") pod \"payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.418980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.418946    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6c415e-bcec-47aa-9ec4-28618957abe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.419198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.419109    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xnhp\" (UniqueName: \"kubernetes.io/projected/7d6c415e-bcec-47aa-9ec4-28618957abe0-kube-api-access-8xnhp\") pod \"payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.419757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.419729    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6c415e-bcec-47aa-9ec4-28618957abe0-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.437903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.437870    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xnhp\" (UniqueName: \"kubernetes.io/projected/7d6c415e-bcec-47aa-9ec4-28618957abe0-kube-api-access-8xnhp\") pod \"payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.576177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.576092    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d71acda9-cc37-46c4-8363-6247e99cd612" path="/var/lib/kubelet/pods/d71acda9-cc37-46c4-8363-6247e99cd612/volumes"
Nov 03 00:33:34.634520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.634490    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:34.707701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.705053    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:34.933537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:34.933507    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"]
Nov 03 00:33:34.933938 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:33:34.933838    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d04140_4088_45ce_818b_8c88a90d3348.slice/crio-8a4d104f34d7f5ae2247effa566cd65a19761af44902a43cb099b4c856eef680 WatchSource:0}: Error finding container 8a4d104f34d7f5ae2247effa566cd65a19761af44902a43cb099b4c856eef680: Status 404 returned error can't find the container with id 8a4d104f34d7f5ae2247effa566cd65a19761af44902a43cb099b4c856eef680
Nov 03 00:33:35.013079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.013050    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"]
Nov 03 00:33:35.018141 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:33:35.018106    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6c415e_bcec_47aa_9ec4_28618957abe0.slice/crio-fbf8c5712cae9e54edb86887bad1bf1a86ee07a383c5b55cbdc34cb15a8314ed WatchSource:0}: Error finding container fbf8c5712cae9e54edb86887bad1bf1a86ee07a383c5b55cbdc34cb15a8314ed: Status 404 returned error can't find the container with id fbf8c5712cae9e54edb86887bad1bf1a86ee07a383c5b55cbdc34cb15a8314ed
Nov 03 00:33:35.727829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.727796    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6" event={"ID":"7d6c415e-bcec-47aa-9ec4-28618957abe0","Type":"ContainerStarted","Data":"e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a"}
Nov 03 00:33:35.728037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.727848    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6" event={"ID":"7d6c415e-bcec-47aa-9ec4-28618957abe0","Type":"ContainerStarted","Data":"fbf8c5712cae9e54edb86887bad1bf1a86ee07a383c5b55cbdc34cb15a8314ed"}
Nov 03 00:33:35.728037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.727875    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:35.729538 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.729514    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg" event={"ID":"56d04140-4088-45ce-818b-8c88a90d3348","Type":"ContainerStarted","Data":"dcaa0a2e0bd6d89f5c748e20f05b666517eee9349007d6fad0c9a54be6cb866b"}
Nov 03 00:33:35.729660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.729555    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg" event={"ID":"56d04140-4088-45ce-818b-8c88a90d3348","Type":"ContainerStarted","Data":"8a4d104f34d7f5ae2247effa566cd65a19761af44902a43cb099b4c856eef680"}
Nov 03 00:33:35.729660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.729575    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:35.766794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.766741    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6" podStartSLOduration=1.766727674 podStartE2EDuration="1.766727674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:35.763380893 +0000 UTC m=+1653.318732930" watchObservedRunningTime="2026-11-03 00:33:35.766727674 +0000 UTC m=+1653.322079697"
Nov 03 00:33:35.801530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:35.801478    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg" podStartSLOduration=2.801462564 podStartE2EDuration="2.801462564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:35.799095672 +0000 UTC m=+1653.354447696" watchObservedRunningTime="2026-11-03 00:33:35.801462564 +0000 UTC m=+1653.356814588"
Nov 03 00:33:36.576655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.576623    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"]
Nov 03 00:33:36.582503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.582480    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.609826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.609784    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-5d51a5-openshift-default-dockercfg-wxxrd\""
Nov 03 00:33:36.613048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.613007    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"]
Nov 03 00:33:36.661695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.661662    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlkrj\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-kube-api-access-vlkrj\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.661863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.661778    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.661863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.661830    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.661863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.661850    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.661977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.661933    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.662053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.662042    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.662090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.662073    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.662214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.662193    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.662317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.662247    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763333    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763377    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763429    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763457    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763503    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlkrj\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-kube-api-access-vlkrj\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763581    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763633    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763659    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763706    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.763987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.763947    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-data\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.764369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.764097    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-certs\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.764369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.764147    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.764369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.764341    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istiod-ca-cert\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.764369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.764351    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-credential-socket\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.766353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.766334    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-envoy\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.766559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.766528    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-podinfo\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.789987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.789960    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-token\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.790309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.790247    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlkrj\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-kube-api-access-vlkrj\") pod \"e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") " pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:36.913908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:36.913881    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:37.229920 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:33:37.229887    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc617f66f_b0c9_4b98_98e7_90ef7ac2a617.slice/crio-eda40aec0a6122891eaa1a15dd16c5a29d951f465c8ac0d294266e7f52b25a6a WatchSource:0}: Error finding container eda40aec0a6122891eaa1a15dd16c5a29d951f465c8ac0d294266e7f52b25a6a: Status 404 returned error can't find the container with id eda40aec0a6122891eaa1a15dd16c5a29d951f465c8ac0d294266e7f52b25a6a
Nov 03 00:33:37.230116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:37.230093    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"]
Nov 03 00:33:37.232620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:37.232592    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:37.232701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:37.232653    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:37.232701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:37.232681    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:33:37.784872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:37.783744    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" event={"ID":"c617f66f-b0c9-4b98-98e7-90ef7ac2a617","Type":"ContainerStarted","Data":"07bda56cc5f8657364e9ffd33d0725344a5dc16b38dd48a787d195aa1cc50610"}
Nov 03 00:33:37.784872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:37.783811    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" event={"ID":"c617f66f-b0c9-4b98-98e7-90ef7ac2a617","Type":"ContainerStarted","Data":"eda40aec0a6122891eaa1a15dd16c5a29d951f465c8ac0d294266e7f52b25a6a"}
Nov 03 00:33:37.924763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:37.924731    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:38.916474 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:38.916442    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 00:33:38.916868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:38.916497    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 00:33:39.914549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:39.914515    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 00:33:39.914960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:39.914922    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 00:33:40.916290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:40.915302    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 00:33:40.916290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:40.915365    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 00:33:41.914828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:41.914794    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 00:33:41.915030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:41.914855    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 00:33:42.914600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:42.914560    6000 patch_prober.go:29] interesting pod/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body=
Nov 03 00:33:42.915028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:42.914623    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused"
Nov 03 00:33:43.918523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:43.918494    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:43.918932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:43.918888    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:43.919552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:43.919535    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:33:43.974082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:43.974036    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" podStartSLOduration=7.974023068 podStartE2EDuration="7.974023068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:33:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:33:37.891623037 +0000 UTC m=+1655.446975062" watchObservedRunningTime="2026-11-03 00:33:43.974023068 +0000 UTC m=+1661.529375091"
Nov 03 00:33:46.740131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:46.740103    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:33:46.740656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:46.740177    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:33:49.896422 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:33:49.896380    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:33:53.437802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:53.437768    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:33:56.332404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:56.332353    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": context deadline exceeded"
Nov 03 00:33:56.364947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:56.364905    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 00:33:56.365565 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:56.365471    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:56.928277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:56.902833    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:33:57.296336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:57.293386    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 00:33:57.381846 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:33:57.381798    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:19.897415 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:19.897381    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:34:20.355480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:20.355447    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"]
Nov 03 00:34:20.379143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:20.379105    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"]
Nov 03 00:34:25.356693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.356650    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx" podUID="c6efa864-b60d-48b0-a1a0-6194a19578eb" containerName="payload-processing" containerID="cri-o://ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914" gracePeriod=25
Nov 03 00:34:25.379825 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.379768    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89" podUID="660ea1a6-b244-4cc2-b867-5812fd2151d2" containerName="payload-pre-processing" containerID="cri-o://345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44" gracePeriod=25
Nov 03 00:34:25.677075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.677044    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:34:25.704534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.704510    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:34:25.774863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.774833    6000 generic.go:356] "Generic (PLEG): container finished" podID="c6efa864-b60d-48b0-a1a0-6194a19578eb" containerID="ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914" exitCode=0
Nov 03 00:34:25.775056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.774882    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"
Nov 03 00:34:25.775056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.774929    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx" event={"ID":"c6efa864-b60d-48b0-a1a0-6194a19578eb","Type":"ContainerDied","Data":"ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914"}
Nov 03 00:34:25.775056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.774985    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx" event={"ID":"c6efa864-b60d-48b0-a1a0-6194a19578eb","Type":"ContainerDied","Data":"30ef9087117376ab196fd4a955c8d60e093855c585c0e8b6dc8e1bb29418d4a2"}
Nov 03 00:34:25.775056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.775015    6000 scope.go:122] "RemoveContainer" containerID="ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914"
Nov 03 00:34:25.776791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.776766    6000 generic.go:356] "Generic (PLEG): container finished" podID="660ea1a6-b244-4cc2-b867-5812fd2151d2" containerID="345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44" exitCode=0
Nov 03 00:34:25.776938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.776812    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89" event={"ID":"660ea1a6-b244-4cc2-b867-5812fd2151d2","Type":"ContainerDied","Data":"345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44"}
Nov 03 00:34:25.776938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.776821    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"
Nov 03 00:34:25.776938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.776851    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89" event={"ID":"660ea1a6-b244-4cc2-b867-5812fd2151d2","Type":"ContainerDied","Data":"57a2a485266fd89fb4250885330fc5bad9d0290c8330a7940151aad65d884221"}
Nov 03 00:34:25.789017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.788993    6000 scope.go:122] "RemoveContainer" containerID="ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914"
Nov 03 00:34:25.789386 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:25.789364    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914\": container with ID starting with ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914 not found: ID does not exist" containerID="ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914"
Nov 03 00:34:25.789511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.789397    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914"} err="failed to get container status \"ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914\": rpc error: code = NotFound desc = could not find container \"ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914\": container with ID starting with ec848e7fa12dd10705acf801b36d7b5aee31238bbd3b9712e9974a005a451914 not found: ID does not exist"
Nov 03 00:34:25.789511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.789423    6000 scope.go:122] "RemoveContainer" containerID="345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44"
Nov 03 00:34:25.799553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.799526    6000 scope.go:122] "RemoveContainer" containerID="345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44"
Nov 03 00:34:25.799792 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:25.799772    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44\": container with ID starting with 345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44 not found: ID does not exist" containerID="345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44"
Nov 03 00:34:25.799859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.799801    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44"} err="failed to get container status \"345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44\": rpc error: code = NotFound desc = could not find container \"345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44\": container with ID starting with 345c4ca183789ed6854c0c93a9747b544332817d0423e2fbb8914b1c8a100e44 not found: ID does not exist"
Nov 03 00:34:25.802275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.802232    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660ea1a6-b244-4cc2-b867-5812fd2151d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ea1a6-b244-4cc2-b867-5812fd2151d2-plugins-config-volume\") pod \"660ea1a6-b244-4cc2-b867-5812fd2151d2\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") "
Nov 03 00:34:25.802389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.802375    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6efa864-b60d-48b0-a1a0-6194a19578eb-kube-api-access-vzrfr\" (UniqueName: \"kubernetes.io/projected/c6efa864-b60d-48b0-a1a0-6194a19578eb-kube-api-access-vzrfr\") pod \"c6efa864-b60d-48b0-a1a0-6194a19578eb\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") "
Nov 03 00:34:25.802583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.802562    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660ea1a6-b244-4cc2-b867-5812fd2151d2-plugins-config-volume" pod "660ea1a6-b244-4cc2-b867-5812fd2151d2" (UID: "660ea1a6-b244-4cc2-b867-5812fd2151d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:25.802852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.802832    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660ea1a6-b244-4cc2-b867-5812fd2151d2-kube-api-access-7b86t\" (UniqueName: \"kubernetes.io/projected/660ea1a6-b244-4cc2-b867-5812fd2151d2-kube-api-access-7b86t\") pod \"660ea1a6-b244-4cc2-b867-5812fd2151d2\" (UID: \"660ea1a6-b244-4cc2-b867-5812fd2151d2\") "
Nov 03 00:34:25.803095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.803078    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6efa864-b60d-48b0-a1a0-6194a19578eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6efa864-b60d-48b0-a1a0-6194a19578eb-plugins-config-volume\") pod \"c6efa864-b60d-48b0-a1a0-6194a19578eb\" (UID: \"c6efa864-b60d-48b0-a1a0-6194a19578eb\") "
Nov 03 00:34:25.803542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.803522    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6efa864-b60d-48b0-a1a0-6194a19578eb-plugins-config-volume" pod "c6efa864-b60d-48b0-a1a0-6194a19578eb" (UID: "c6efa864-b60d-48b0-a1a0-6194a19578eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:25.803650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.803638    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6efa864-b60d-48b0-a1a0-6194a19578eb-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:25.803705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.803653    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/660ea1a6-b244-4cc2-b867-5812fd2151d2-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:25.804707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.804689    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6efa864-b60d-48b0-a1a0-6194a19578eb-kube-api-access-vzrfr" pod "c6efa864-b60d-48b0-a1a0-6194a19578eb" (UID: "c6efa864-b60d-48b0-a1a0-6194a19578eb"). InnerVolumeSpecName "kube-api-access-vzrfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:25.804921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.804903    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660ea1a6-b244-4cc2-b867-5812fd2151d2-kube-api-access-7b86t" pod "660ea1a6-b244-4cc2-b867-5812fd2151d2" (UID: "660ea1a6-b244-4cc2-b867-5812fd2151d2"). InnerVolumeSpecName "kube-api-access-7b86t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:25.913334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.912311    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzrfr\" (UniqueName: \"kubernetes.io/projected/c6efa864-b60d-48b0-a1a0-6194a19578eb-kube-api-access-vzrfr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:25.913334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:25.912352    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b86t\" (UniqueName: \"kubernetes.io/projected/660ea1a6-b244-4cc2-b867-5812fd2151d2-kube-api-access-7b86t\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:26.131067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:26.131014    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"]
Nov 03 00:34:26.142071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:26.142045    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx"]
Nov 03 00:34:26.175548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:26.175422    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"]
Nov 03 00:34:26.179028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:26.178997    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-5d51a5-55b486f479-fbj89"]
Nov 03 00:34:26.567240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:26.567162    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660ea1a6-b244-4cc2-b867-5812fd2151d2" path="/var/lib/kubelet/pods/660ea1a6-b244-4cc2-b867-5812fd2151d2/volumes"
Nov 03 00:34:26.567641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:26.567622    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6efa864-b60d-48b0-a1a0-6194a19578eb" path="/var/lib/kubelet/pods/c6efa864-b60d-48b0-a1a0-6194a19578eb/volumes"
Nov 03 00:34:26.656370 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:26.656332    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-5d51a5-5f49d544cc-2q8fx" podUID="c6efa864-b60d-48b0-a1a0-6194a19578eb" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.136:50052\" within 1s: context deadline exceeded"
Nov 03 00:34:26.656544 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:34:26.656425    6000 logging.go:55] [core] [Channel #1427 SubChannel #1428]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.136:50052", ServerName: "10.134.0.136:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.136:50052: operation was canceled"
Nov 03 00:34:29.116548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.116513    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"]
Nov 03 00:34:29.117021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.116737    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg" podUID="7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" containerName="maas-api" containerID="cri-o://a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715" gracePeriod=30
Nov 03 00:34:29.448013 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.447985    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:34:29.571469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.571440    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-maas-api-tls\") pod \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") "
Nov 03 00:34:29.571672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.571600    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-metrics-certs\") pod \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") "
Nov 03 00:34:29.571672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.571635    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-kube-api-access-zpk8l\" (UniqueName: \"kubernetes.io/projected/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-kube-api-access-zpk8l\") pod \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") "
Nov 03 00:34:29.571806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.571690    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-openshift-service-ca\") pod \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\" (UID: \"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca\") "
Nov 03 00:34:29.572177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.572147    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-openshift-service-ca" pod "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" (UID: "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:29.572782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.572757    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:29.573970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.573940    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-metrics-certs" pod "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" (UID: "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:29.574085 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.574045    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-maas-api-tls" pod "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" (UID: "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:29.574142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.574086    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-kube-api-access-zpk8l" pod "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" (UID: "7e879dc1-80a5-47c3-a3e1-8108b64dc9ca"). InnerVolumeSpecName "kube-api-access-zpk8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:29.677399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.675733    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:29.677399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.675770    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zpk8l\" (UniqueName: \"kubernetes.io/projected/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-kube-api-access-zpk8l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:29.677399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.675784    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:29.830317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.829457    6000 generic.go:356] "Generic (PLEG): container finished" podID="7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" containerID="a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715" exitCode=0
Nov 03 00:34:29.830317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.829523    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"
Nov 03 00:34:29.830317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.829642    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg" event={"ID":"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca","Type":"ContainerDied","Data":"a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715"}
Nov 03 00:34:29.830317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.829693    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg" event={"ID":"7e879dc1-80a5-47c3-a3e1-8108b64dc9ca","Type":"ContainerDied","Data":"81d56e7dfaa30809a9326877088c4f808c02e4189d3752ce863e9d03405cd6ab"}
Nov 03 00:34:29.830317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.829723    6000 scope.go:122] "RemoveContainer" containerID="a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715"
Nov 03 00:34:29.842198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.842175    6000 scope.go:122] "RemoveContainer" containerID="a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715"
Nov 03 00:34:29.842496 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:29.842471    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715\": container with ID starting with a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715 not found: ID does not exist" containerID="a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715"
Nov 03 00:34:29.842583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.842503    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715"} err="failed to get container status \"a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715\": rpc error: code = NotFound desc = could not find container \"a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715\": container with ID starting with a5b3befdeefccb2e1d3641f000f8277c6c8710bc4a6e1681d29e3e49d7d39715 not found: ID does not exist"
Nov 03 00:34:29.880171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.880144    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"]
Nov 03 00:34:29.891241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:29.891216    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-2af7b0-679b97d55c-6npgg"]
Nov 03 00:34:30.568694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:30.568658    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" path="/var/lib/kubelet/pods/7e879dc1-80a5-47c3-a3e1-8108b64dc9ca/volumes"
Nov 03 00:34:31.440223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.440188    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"]
Nov 03 00:34:31.440449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.440410    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh" podUID="3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" containerName="maas-api" containerID="cri-o://1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206" gracePeriod=30
Nov 03 00:34:31.798278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.794705    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:34:31.830065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.828558    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-openshift-service-ca\") pod \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") "
Nov 03 00:34:31.830065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.828740    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-kube-api-access-lm9m2\" (UniqueName: \"kubernetes.io/projected/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-kube-api-access-lm9m2\") pod \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") "
Nov 03 00:34:31.830065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.828813    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-maas-api-tls\") pod \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") "
Nov 03 00:34:31.830065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.828920    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-metrics-certs\") pod \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\" (UID: \"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d\") "
Nov 03 00:34:31.832107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.832046    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-openshift-service-ca" pod "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" (UID: "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:31.849116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.846656    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-metrics-certs" pod "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" (UID: "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:31.866499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.853133    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-kube-api-access-lm9m2" pod "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" (UID: "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d"). InnerVolumeSpecName "kube-api-access-lm9m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:31.866499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.860794    6000 generic.go:356] "Generic (PLEG): container finished" podID="3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" containerID="1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206" exitCode=0
Nov 03 00:34:31.866499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.860876    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh" event={"ID":"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d","Type":"ContainerDied","Data":"1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206"}
Nov 03 00:34:31.866499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.860924    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh" event={"ID":"3b151cdb-1789-40cb-bc0c-c6d7f75eef2d","Type":"ContainerDied","Data":"0b8224407aca690c627ee677a2a6915569df942ca3ed0e4e4e42dffd5bbdf278"}
Nov 03 00:34:31.866499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.860952    6000 scope.go:122] "RemoveContainer" containerID="1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206"
Nov 03 00:34:31.866499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.861092    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"
Nov 03 00:34:31.878239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.878081    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-maas-api-tls" pod "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" (UID: "3b151cdb-1789-40cb-bc0c-c6d7f75eef2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:34:31.891884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.891376    6000 scope.go:122] "RemoveContainer" containerID="1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206"
Nov 03 00:34:31.891884 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:31.891791    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206\": container with ID starting with 1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206 not found: ID does not exist" containerID="1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206"
Nov 03 00:34:31.891884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.891824    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206"} err="failed to get container status \"1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206\": rpc error: code = NotFound desc = could not find container \"1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206\": container with ID starting with 1752ce2b5391b37a45379f3cda3efdcdc1b3c4e8e7328b4d9c0181feffb9a206 not found: ID does not exist"
Nov 03 00:34:31.930529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.930495    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:31.930740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.930714    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:31.930807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.930741    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm9m2\" (UniqueName: \"kubernetes.io/projected/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-kube-api-access-lm9m2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:31.930807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:31.930756    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:32.118953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:32.118915    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"]
Nov 03 00:34:32.119211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:32.119164    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" containerID="cri-o://9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0" gracePeriod=30
Nov 03 00:34:32.330276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:32.329585    6000 patch_prober.go:29] interesting pod/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:34:32.330276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:32.329672    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:34:32.417548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:32.417512    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"]
Nov 03 00:34:32.420542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:32.420514    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-praxis-a-5d51a5-84db6c5cf7-48grh"]
Nov 03 00:34:32.569813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:32.569776    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" path="/var/lib/kubelet/pods/3b151cdb-1789-40cb-bc0c-c6d7f75eef2d/volumes"
Nov 03 00:34:34.314219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:34.314177    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"]
Nov 03 00:34:34.314770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:34.314471    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy" containerID="cri-o://07bda56cc5f8657364e9ffd33d0725344a5dc16b38dd48a787d195aa1cc50610" gracePeriod=30
Nov 03 00:34:35.785118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.785080    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"]
Nov 03 00:34:35.785913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.785891    6000 cpu_manager.go:414] "containerMap: removing container" podUID="3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" containerName="maas-api"
Nov 03 00:34:35.786018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.785915    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" containerName="maas-api"
Nov 03 00:34:35.786018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.785997    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" containerName="maas-api"
Nov 03 00:34:35.786018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786006    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" containerName="maas-api"
Nov 03 00:34:35.786176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786030    6000 cpu_manager.go:414] "containerMap: removing container" podUID="660ea1a6-b244-4cc2-b867-5812fd2151d2" containerName="payload-pre-processing"
Nov 03 00:34:35.786176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786038    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660ea1a6-b244-4cc2-b867-5812fd2151d2" containerName="payload-pre-processing"
Nov 03 00:34:35.786176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786053    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c6efa864-b60d-48b0-a1a0-6194a19578eb" containerName="payload-processing"
Nov 03 00:34:35.786176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786060    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6efa864-b60d-48b0-a1a0-6194a19578eb" containerName="payload-processing"
Nov 03 00:34:35.786395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786191    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6efa864-b60d-48b0-a1a0-6194a19578eb" containerName="payload-processing"
Nov 03 00:34:35.786395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786207    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b151cdb-1789-40cb-bc0c-c6d7f75eef2d" containerName="maas-api"
Nov 03 00:34:35.786395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786231    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e879dc1-80a5-47c3-a3e1-8108b64dc9ca" containerName="maas-api"
Nov 03 00:34:35.786395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.786243    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="660ea1a6-b244-4cc2-b867-5812fd2151d2" containerName="payload-pre-processing"
Nov 03 00:34:35.791342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.791321    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.812363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.810195    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"]
Nov 03 00:34:35.898152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898114    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898234    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898342    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898400    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898518    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898551    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898608    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898656    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.898718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.898712    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr2qj\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-kube-api-access-fr2qj\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.999901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.999867    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:35.999901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.999906    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:35.999970    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr2qj\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-kube-api-access-fr2qj\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000023    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000075    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000119    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000168    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000243    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000284    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000312    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000397    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-data\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000539    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-credential-socket\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000589    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-certs\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.000981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.000958    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istiod-ca-cert\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.002637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.002613    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-envoy\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.002760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.002741    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-podinfo\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.019199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.019164    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-token\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.019334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.019223    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr2qj\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-kube-api-access-fr2qj\") pod \"e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") " pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.109771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.109736    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:36.430861 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:34:36.430825    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4f1e4dd_5085_4b91_91af_cdffe80186c0.slice/crio-f0525efa29ab9fbce9b0024447892c3551926893bc26673aa2d2777d463e5281 WatchSource:0}: Error finding container f0525efa29ab9fbce9b0024447892c3551926893bc26673aa2d2777d463e5281: Status 404 returned error can't find the container with id f0525efa29ab9fbce9b0024447892c3551926893bc26673aa2d2777d463e5281
Nov 03 00:34:36.430995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.430921    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"]
Nov 03 00:34:36.433242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.433207    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:34:36.433396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.433306    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:34:36.433396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.433352    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:34:36.897789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.897752    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" event={"ID":"b4f1e4dd-5085-4b91-91af-cdffe80186c0","Type":"ContainerStarted","Data":"44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b"}
Nov 03 00:34:36.898303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.897814    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" event={"ID":"b4f1e4dd-5085-4b91-91af-cdffe80186c0","Type":"ContainerStarted","Data":"f0525efa29ab9fbce9b0024447892c3551926893bc26673aa2d2777d463e5281"}
Nov 03 00:34:36.960930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:36.960870    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podStartSLOduration=1.9608568210000001 podStartE2EDuration="1.960856821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:36.948446559 +0000 UTC m=+1714.503798595" watchObservedRunningTime="2026-11-03 00:34:36.960856821 +0000 UTC m=+1714.516208844"
Nov 03 00:34:37.110918 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.110877    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:37.112655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.112629    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:37.112787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.112682    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:37.453012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.452979    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:34:37.637163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637122    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-credential-socket\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637194    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-podinfo\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637235    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-kube-api-access-cczlz\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-kube-api-access-cczlz\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637291    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-certs\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637423    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-credential-socket" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:37.637537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637439    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-socket\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637490    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-envoy\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637526    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-token\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-token\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637563    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istiod-ca-cert\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637620    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-certs" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:37.637746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637632    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-data\") pod \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\" (UID: \"9e160351-b07e-4e4d-a7ae-e7096a3912a3\") "
Nov 03 00:34:37.637746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.637707    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-socket" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:37.638022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.638001    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istiod-ca-cert" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:37.638139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.638040    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-data" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:37.638541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.638513    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.638626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.638544    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.638626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.638561    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.638626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.638576    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.638626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.638591    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.639782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.639758    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-podinfo" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:34:37.639903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.639879    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-kube-api-access-cczlz" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "kube-api-access-cczlz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:37.640019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.639998    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-token" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:37.640399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.640373    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-envoy" pod "9e160351-b07e-4e4d-a7ae-e7096a3912a3" (UID: "9e160351-b07e-4e4d-a7ae-e7096a3912a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:37.740280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.740189    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.740443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.740303    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.740443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.740326    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9e160351-b07e-4e4d-a7ae-e7096a3912a3-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.740443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.740341    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cczlz\" (UniqueName: \"kubernetes.io/projected/9e160351-b07e-4e4d-a7ae-e7096a3912a3-kube-api-access-cczlz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:37.919337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.919185    6000 generic.go:356] "Generic (PLEG): container finished" podID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerID="9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0" exitCode=0
Nov 03 00:34:37.919337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.919283    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"
Nov 03 00:34:37.919337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.919306    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" event={"ID":"9e160351-b07e-4e4d-a7ae-e7096a3912a3","Type":"ContainerDied","Data":"9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0"}
Nov 03 00:34:37.919958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.919374    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll" event={"ID":"9e160351-b07e-4e4d-a7ae-e7096a3912a3","Type":"ContainerDied","Data":"0f04638cf68c1adf87bc564855dad7af30c19809a575ecd3e104b16f9cc57a87"}
Nov 03 00:34:37.919958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.919398    6000 scope.go:122] "RemoveContainer" containerID="9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0"
Nov 03 00:34:37.930504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.930486    6000 scope.go:122] "RemoveContainer" containerID="a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746"
Nov 03 00:34:37.940401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.940379    6000 scope.go:122] "RemoveContainer" containerID="9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0"
Nov 03 00:34:37.940660 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:37.940638    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0\": container with ID starting with 9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0 not found: ID does not exist" containerID="9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0"
Nov 03 00:34:37.940718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.940669    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0"} err="failed to get container status \"9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0\": rpc error: code = NotFound desc = could not find container \"9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0\": container with ID starting with 9d9ef5870ad7dc9ef241d21daa5ca30b4d859e67a433451390ecedd78c8fceb0 not found: ID does not exist"
Nov 03 00:34:37.940718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.940684    6000 scope.go:122] "RemoveContainer" containerID="a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746"
Nov 03 00:34:37.940916 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:37.940899    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746\": container with ID starting with a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746 not found: ID does not exist" containerID="a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746"
Nov 03 00:34:37.940958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.940925    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746"} err="failed to get container status \"a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746\": rpc error: code = NotFound desc = could not find container \"a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746\": container with ID starting with a199b7c4a6cc2e1c2f4fd71fd2a0039d4ce1394cb45e67bfa1339824040e5746 not found: ID does not exist"
Nov 03 00:34:37.972042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.972015    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"]
Nov 03 00:34:37.983523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:37.983496    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-2af7b0-openshift-default-6956784c49-whkll"]
Nov 03 00:34:38.112170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:38.111679    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:38.112170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:38.111766    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:38.252934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:38.252904    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:34:38.568611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:38.568535    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" path="/var/lib/kubelet/pods/9e160351-b07e-4e4d-a7ae-e7096a3912a3/volumes"
Nov 03 00:34:39.113279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:39.110689    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:39.113279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:39.110753    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:40.133097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.133057    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:40.133685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.133648    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:40.372945 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.362030    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:40.581125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.580993    6000 generic.go:356] "Generic (PLEG): container finished" podID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerID="07bda56cc5f8657364e9ffd33d0725344a5dc16b38dd48a787d195aa1cc50610" exitCode=0
Nov 03 00:34:40.735301 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.709854    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" event={"ID":"c617f66f-b0c9-4b98-98e7-90ef7ac2a617","Type":"ContainerDied","Data":"07bda56cc5f8657364e9ffd33d0725344a5dc16b38dd48a787d195aa1cc50610"}
Nov 03 00:34:40.906859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.906810    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:34:40.991574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.991481    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-envoy\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:40.991923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:40.991900    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-kube-api-access-vlkrj\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-kube-api-access-vlkrj\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.012375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.012341    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-certs\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.012658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.012639    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-data\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.012844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.012826    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-credential-socket\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.013038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.013020    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istiod-ca-cert\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.013198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.013183    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-token\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-token\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.013452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.013432    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-podinfo\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.013657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.013639    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-socket\") pod \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\" (UID: \"c617f66f-b0c9-4b98-98e7-90ef7ac2a617\") "
Nov 03 00:34:41.034795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.029603    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istiod-ca-cert" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:41.034795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.029886    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-data" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.034795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.030104    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-credential-socket" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.034795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.034507    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-socket" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.035248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.035223    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-certs" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.119742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.119709    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.119968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.119953    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.120213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.120116    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.120213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.120136    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.120213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.120151    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.120903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.120879    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-token" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:41.147761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.147704    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-kube-api-access-vlkrj" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "kube-api-access-vlkrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:41.148361 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.148337    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:41.160328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.148527    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:41.219448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.219386    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-podinfo" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:34:41.241291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.221385    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-envoy" pod "c617f66f-b0c9-4b98-98e7-90ef7ac2a617" (UID: "c617f66f-b0c9-4b98-98e7-90ef7ac2a617"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:34:41.241291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.222595    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.241291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.222622    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.241291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.222639    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.241291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.222653    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlkrj\" (UniqueName: \"kubernetes.io/projected/c617f66f-b0c9-4b98-98e7-90ef7ac2a617-kube-api-access-vlkrj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:41.343552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.342624    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.376280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.364465    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.376280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.369971    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.376280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.370786    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 00:34:41.376280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.371082    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:34:41.376657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.376377    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:34:41.867373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.867328    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:41.881742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.881705    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk" event={"ID":"c617f66f-b0c9-4b98-98e7-90ef7ac2a617","Type":"ContainerDied","Data":"eda40aec0a6122891eaa1a15dd16c5a29d951f465c8ac0d294266e7f52b25a6a"}
Nov 03 00:34:41.882023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.882005    6000 scope.go:122] "RemoveContainer" containerID="07bda56cc5f8657364e9ffd33d0725344a5dc16b38dd48a787d195aa1cc50610"
Nov 03 00:34:41.882306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:41.882287    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"
Nov 03 00:34:42.116358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.116316    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"]
Nov 03 00:34:42.120283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.117691    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:42.145543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.145503    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:42.233410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.233368    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:42.234050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.234018    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.391455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.391352    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.417280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.399456    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.443426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.443375    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.501284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.470071    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:34:42.567284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.542382    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:34:42.567627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.567601    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:34:42.649997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.406042    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 00:34:42.649997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.620331    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:42.649997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.620404    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:42.649997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.634215    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Nov 03 00:34:42.784432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.784386    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded"
Nov 03 00:34:42.903291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.901332    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:34:43.010812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.968208    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:43.010812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:42.968316    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.249351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.218597    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:34:43.250467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.250440    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"]
Nov 03 00:34:43.258283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.229308    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:43.258511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.258480    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:43.444239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.410120    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.444908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.444875    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.472309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.471857    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.605279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.602367    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:43.745622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.709493    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"]
Nov 03 00:34:44.079369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.872427    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-5d51a5-openshift-default-968d79677-8gpvk"]
Nov 03 00:34:44.079369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:43.908163    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:44.107343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.106471    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:34:44.109540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.109520    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:34:44.111964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.111915    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:44.112195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.112173    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:44.132974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.132944    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:34:44.268787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.268758    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:34:44.285746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.285718    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:34:44.567308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:44.567279    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" path="/var/lib/kubelet/pods/c617f66f-b0c9-4b98-98e7-90ef7ac2a617/volumes"
Nov 03 00:34:45.114043 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:45.111667    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:45.114043 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:45.111748    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:46.110374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:46.110333    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body=
Nov 03 00:34:46.110853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:46.110410    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused"
Nov 03 00:34:47.117585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.117542    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg" podUID="56d04140-4088-45ce-818b-8c88a90d3348" containerName="payload-processing" containerID="cri-o://dcaa0a2e0bd6d89f5c748e20f05b666517eee9349007d6fad0c9a54be6cb866b" gracePeriod=25
Nov 03 00:34:47.259167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.259131    6000 generic.go:356] "Generic (PLEG): container finished" podID="56d04140-4088-45ce-818b-8c88a90d3348" containerID="dcaa0a2e0bd6d89f5c748e20f05b666517eee9349007d6fad0c9a54be6cb866b" exitCode=0
Nov 03 00:34:47.259321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.259271    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg" event={"ID":"56d04140-4088-45ce-818b-8c88a90d3348","Type":"ContainerDied","Data":"dcaa0a2e0bd6d89f5c748e20f05b666517eee9349007d6fad0c9a54be6cb866b"}
Nov 03 00:34:47.398878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.398848    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:34:47.506134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.506102    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"]
Nov 03 00:34:47.507381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507353    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy"
Nov 03 00:34:47.507381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507376    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy"
Nov 03 00:34:47.507579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507433    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy"
Nov 03 00:34:47.507579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507442    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy"
Nov 03 00:34:47.507579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507462    6000 cpu_manager.go:414] "containerMap: removing container" podUID="56d04140-4088-45ce-818b-8c88a90d3348" containerName="payload-processing"
Nov 03 00:34:47.507579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507469    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d04140-4088-45ce-818b-8c88a90d3348" containerName="payload-processing"
Nov 03 00:34:47.507579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507484    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy"
Nov 03 00:34:47.507579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507492    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy"
Nov 03 00:34:47.507869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507649    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c617f66f-b0c9-4b98-98e7-90ef7ac2a617" containerName="istio-proxy"
Nov 03 00:34:47.507869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507665    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d04140-4088-45ce-818b-8c88a90d3348" containerName="payload-processing"
Nov 03 00:34:47.507869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507677    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy"
Nov 03 00:34:47.507869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.507692    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e160351-b07e-4e4d-a7ae-e7096a3912a3" containerName="istio-proxy"
Nov 03 00:34:47.511503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.511479    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.512946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.512918    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d04140-4088-45ce-818b-8c88a90d3348-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d04140-4088-45ce-818b-8c88a90d3348-plugins-config-volume\") pod \"56d04140-4088-45ce-818b-8c88a90d3348\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") "
Nov 03 00:34:47.513184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.513161    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d04140-4088-45ce-818b-8c88a90d3348-kube-api-access-qbzn2\" (UniqueName: \"kubernetes.io/projected/56d04140-4088-45ce-818b-8c88a90d3348-kube-api-access-qbzn2\") pod \"56d04140-4088-45ce-818b-8c88a90d3348\" (UID: \"56d04140-4088-45ce-818b-8c88a90d3348\") "
Nov 03 00:34:47.513966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.513942    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d04140-4088-45ce-818b-8c88a90d3348-plugins-config-volume" pod "56d04140-4088-45ce-818b-8c88a90d3348" (UID: "56d04140-4088-45ce-818b-8c88a90d3348"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:47.517539 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.517505    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d04140-4088-45ce-818b-8c88a90d3348-kube-api-access-qbzn2" pod "56d04140-4088-45ce-818b-8c88a90d3348" (UID: "56d04140-4088-45ce-818b-8c88a90d3348"). InnerVolumeSpecName "kube-api-access-qbzn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:47.528309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.528271    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-065de0-openshift-default-dockercfg-hvngn\""
Nov 03 00:34:47.567337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.567301    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"]
Nov 03 00:34:47.614877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.614825    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-certs\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.614956    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-data\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615025    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-podinfo\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615084    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0155e021-c45b-4fc5-a94e-a2df797f965f-istiod-ca-cert\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615129    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fg5j\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-kube-api-access-5fg5j\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615235    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-credential-socket\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615303    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-socket\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615356    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-token\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615398    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-envoy\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.615530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615489    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbzn2\" (UniqueName: \"kubernetes.io/projected/56d04140-4088-45ce-818b-8c88a90d3348-kube-api-access-qbzn2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:47.615530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.615506    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/56d04140-4088-45ce-818b-8c88a90d3348-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.740945    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-credential-socket\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741122    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-socket\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741184    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-token\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741226    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-envoy\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741302    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-certs\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741407    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-data\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741474    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-podinfo\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741519    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0155e021-c45b-4fc5-a94e-a2df797f965f-istiod-ca-cert\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.756004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.741565    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fg5j\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-kube-api-access-5fg5j\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.788283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.784673    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-socket\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.788283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.785910    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fg5j\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-kube-api-access-5fg5j\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.788283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.786043    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-credential-socket\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.788283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.786590    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-data\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.795266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.794436    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0155e021-c45b-4fc5-a94e-a2df797f965f-istiod-ca-cert\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.802532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.802504    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-certs\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.817090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.815578    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-podinfo\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.817777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.817722    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-envoy\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.820578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.820553    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-token\") pod \"e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") " pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:47.879307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:47.878083    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:48.179768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.151431    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:34:48.179768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.151554    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:34:48.251768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.251729    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6" podUID="7d6c415e-bcec-47aa-9ec4-28618957abe0" containerName="payload-pre-processing" containerID="cri-o://e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a" gracePeriod=25
Nov 03 00:34:48.267288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.267226    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg" event={"ID":"56d04140-4088-45ce-818b-8c88a90d3348","Type":"ContainerDied","Data":"8a4d104f34d7f5ae2247effa566cd65a19761af44902a43cb099b4c856eef680"}
Nov 03 00:34:48.267416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.267303    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"
Nov 03 00:34:48.267416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.267344    6000 scope.go:122] "RemoveContainer" containerID="dcaa0a2e0bd6d89f5c748e20f05b666517eee9349007d6fad0c9a54be6cb866b"
Nov 03 00:34:48.302693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.302664    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"]
Nov 03 00:34:48.320941 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:34:48.320913    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0155e021_c45b_4fc5_a94e_a2df797f965f.slice/crio-854a0d8a32096a5a3cfca1de414885f53c8eddc15353deaf0f827984349a0538 WatchSource:0}: Error finding container 854a0d8a32096a5a3cfca1de414885f53c8eddc15353deaf0f827984349a0538: Status 404 returned error can't find the container with id 854a0d8a32096a5a3cfca1de414885f53c8eddc15353deaf0f827984349a0538
Nov 03 00:34:48.359177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.359134    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:34:48.359377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.359231    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:34:48.359377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.359296    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:34:48.365441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.365416    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"]
Nov 03 00:34:48.369267 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.369237    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-a19e67-d77bbd859-sgphg"]
Nov 03 00:34:48.531137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.531101    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:48.532815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.532785    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:48.535641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.535615    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:34:48.560472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.560404    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:34:48.571057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.571022    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d04140-4088-45ce-818b-8c88a90d3348" path="/var/lib/kubelet/pods/56d04140-4088-45ce-818b-8c88a90d3348/volumes"
Nov 03 00:34:48.683893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.683853    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6c415e-bcec-47aa-9ec4-28618957abe0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6c415e-bcec-47aa-9ec4-28618957abe0-plugins-config-volume\") pod \"7d6c415e-bcec-47aa-9ec4-28618957abe0\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") "
Nov 03 00:34:48.684068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.684027    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6c415e-bcec-47aa-9ec4-28618957abe0-kube-api-access-8xnhp\" (UniqueName: \"kubernetes.io/projected/7d6c415e-bcec-47aa-9ec4-28618957abe0-kube-api-access-8xnhp\") pod \"7d6c415e-bcec-47aa-9ec4-28618957abe0\" (UID: \"7d6c415e-bcec-47aa-9ec4-28618957abe0\") "
Nov 03 00:34:48.685413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.685383    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6c415e-bcec-47aa-9ec4-28618957abe0-plugins-config-volume" pod "7d6c415e-bcec-47aa-9ec4-28618957abe0" (UID: "7d6c415e-bcec-47aa-9ec4-28618957abe0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:34:48.687698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.687660    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6c415e-bcec-47aa-9ec4-28618957abe0-kube-api-access-8xnhp" pod "7d6c415e-bcec-47aa-9ec4-28618957abe0" (UID: "7d6c415e-bcec-47aa-9ec4-28618957abe0"). InnerVolumeSpecName "kube-api-access-8xnhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:34:48.786632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.786594    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xnhp\" (UniqueName: \"kubernetes.io/projected/7d6c415e-bcec-47aa-9ec4-28618957abe0-kube-api-access-8xnhp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:48.786632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:48.786627    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d6c415e-bcec-47aa-9ec4-28618957abe0-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:34:49.276584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.276540    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" event={"ID":"0155e021-c45b-4fc5-a94e-a2df797f965f","Type":"ContainerStarted","Data":"352aba70745a9a63c4bd6b014a199fad79ec4cb612410abf2721e06f7ad85fd5"}
Nov 03 00:34:49.277032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.276608    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" event={"ID":"0155e021-c45b-4fc5-a94e-a2df797f965f","Type":"ContainerStarted","Data":"854a0d8a32096a5a3cfca1de414885f53c8eddc15353deaf0f827984349a0538"}
Nov 03 00:34:49.278447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.278414    6000 generic.go:356] "Generic (PLEG): container finished" podID="7d6c415e-bcec-47aa-9ec4-28618957abe0" containerID="e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a" exitCode=0
Nov 03 00:34:49.278693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.278462    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"
Nov 03 00:34:49.278693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.278491    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6" event={"ID":"7d6c415e-bcec-47aa-9ec4-28618957abe0","Type":"ContainerDied","Data":"e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a"}
Nov 03 00:34:49.278693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.278544    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6" event={"ID":"7d6c415e-bcec-47aa-9ec4-28618957abe0","Type":"ContainerDied","Data":"fbf8c5712cae9e54edb86887bad1bf1a86ee07a383c5b55cbdc34cb15a8314ed"}
Nov 03 00:34:49.278693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.278573    6000 scope.go:122] "RemoveContainer" containerID="e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a"
Nov 03 00:34:49.291171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.291153    6000 scope.go:122] "RemoveContainer" containerID="e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a"
Nov 03 00:34:49.291513 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:49.291490    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a\": container with ID starting with e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a not found: ID does not exist" containerID="e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a"
Nov 03 00:34:49.291624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.291520    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a"} err="failed to get container status \"e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a\": rpc error: code = NotFound desc = could not find container \"e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a\": container with ID starting with e9ae0bd25ea42395359fca30822c304f8df9a1d3d83dd9bc3d1100b0d939d71a not found: ID does not exist"
Nov 03 00:34:49.319341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.319285    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" podStartSLOduration=2.319251098 podStartE2EDuration="2.319251098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:49.316642184 +0000 UTC m=+1726.871994219" watchObservedRunningTime="2026-11-03 00:34:49.319251098 +0000 UTC m=+1726.874618065"
Nov 03 00:34:49.352564 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.352533    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"]
Nov 03 00:34:49.357715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.357686    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-a19e67-575c997c9-8gzb6"]
Nov 03 00:34:49.892313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.890970    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:49.917281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:49.915779    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:49.953505 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:34:49.953469    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:34:50.290742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:50.290645    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:50.291739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:50.291721    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:34:50.571480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:50.571389    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d6c415e-bcec-47aa-9ec4-28618957abe0" path="/var/lib/kubelet/pods/7d6c415e-bcec-47aa-9ec4-28618957abe0/volumes"
Nov 03 00:34:53.677793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.677765    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"]
Nov 03 00:34:53.678422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.678407    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7d6c415e-bcec-47aa-9ec4-28618957abe0" containerName="payload-pre-processing"
Nov 03 00:34:53.678485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.678423    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6c415e-bcec-47aa-9ec4-28618957abe0" containerName="payload-pre-processing"
Nov 03 00:34:53.678540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.678531    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6c415e-bcec-47aa-9ec4-28618957abe0" containerName="payload-pre-processing"
Nov 03 00:34:53.681854 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.681838    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:53.700375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.700347    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"]
Nov 03 00:34:53.709482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.709458    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-065de0\""
Nov 03 00:34:53.741284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.740706    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-065de0-dockercfg-fmn56\""
Nov 03 00:34:53.897321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.891600    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45eccb1-a80a-4972-9293-b569f843f19f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:53.897321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.891847    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bg72l\" (UniqueName: \"kubernetes.io/projected/b45eccb1-a80a-4972-9293-b569f843f19f-kube-api-access-bg72l\") pod \"payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:53.958094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.958024    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"]
Nov 03 00:34:53.962219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.962200    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:53.969172 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.969152    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"]
Nov 03 00:34:53.992872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.992834    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bg72l\" (UniqueName: \"kubernetes.io/projected/b45eccb1-a80a-4972-9293-b569f843f19f-kube-api-access-bg72l\") pod \"payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:53.993033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.992922    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:53.993033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.992969    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8v2k\" (UniqueName: \"kubernetes.io/projected/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-kube-api-access-m8v2k\") pod \"payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:53.993137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.993099    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45eccb1-a80a-4972-9293-b569f843f19f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:53.993972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:53.993948    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45eccb1-a80a-4972-9293-b569f843f19f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:54.011811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.011779    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bg72l\" (UniqueName: \"kubernetes.io/projected/b45eccb1-a80a-4972-9293-b569f843f19f-kube-api-access-bg72l\") pod \"payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:54.094966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.094929    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:54.095175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.094986    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8v2k\" (UniqueName: \"kubernetes.io/projected/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-kube-api-access-m8v2k\") pod \"payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:54.095740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.095720    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:54.113504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.113480    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8v2k\" (UniqueName: \"kubernetes.io/projected/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-kube-api-access-m8v2k\") pod \"payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:54.278968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.278881    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:54.295672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.295646    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:54.581511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.581477    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"]
Nov 03 00:34:54.581890 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:34:54.581866    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe2e57d_7713_4ca2_afb5_14c11fd739f9.slice/crio-37cccc0e69fa643f10306d80a2a5f7f5519579a58cd50eb3ff4a261f44c8740f WatchSource:0}: Error finding container 37cccc0e69fa643f10306d80a2a5f7f5519579a58cd50eb3ff4a261f44c8740f: Status 404 returned error can't find the container with id 37cccc0e69fa643f10306d80a2a5f7f5519579a58cd50eb3ff4a261f44c8740f
Nov 03 00:34:54.608186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:54.608158    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"]
Nov 03 00:34:54.616598 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:34:54.616567    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45eccb1_a80a_4972_9293_b569f843f19f.slice/crio-d8601187f96b3a81ddddf8aaa33866838a9dc7b03b36bffe3c6acf596ff23ca7 WatchSource:0}: Error finding container d8601187f96b3a81ddddf8aaa33866838a9dc7b03b36bffe3c6acf596ff23ca7: Status 404 returned error can't find the container with id d8601187f96b3a81ddddf8aaa33866838a9dc7b03b36bffe3c6acf596ff23ca7
Nov 03 00:34:55.325101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.325062    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" event={"ID":"b45eccb1-a80a-4972-9293-b569f843f19f","Type":"ContainerStarted","Data":"5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f"}
Nov 03 00:34:55.325634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.325284    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:34:55.325634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.325332    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" event={"ID":"b45eccb1-a80a-4972-9293-b569f843f19f","Type":"ContainerStarted","Data":"d8601187f96b3a81ddddf8aaa33866838a9dc7b03b36bffe3c6acf596ff23ca7"}
Nov 03 00:34:55.326786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.326758    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" event={"ID":"bfe2e57d-7713-4ca2-afb5-14c11fd739f9","Type":"ContainerStarted","Data":"7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb"}
Nov 03 00:34:55.326862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.326807    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" event={"ID":"bfe2e57d-7713-4ca2-afb5-14c11fd739f9","Type":"ContainerStarted","Data":"37cccc0e69fa643f10306d80a2a5f7f5519579a58cd50eb3ff4a261f44c8740f"}
Nov 03 00:34:55.326862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.326833    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:34:55.363062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.362769    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" podStartSLOduration=2.362752537 podStartE2EDuration="2.362752537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:55.362509109 +0000 UTC m=+1732.917861135" watchObservedRunningTime="2026-11-03 00:34:55.362752537 +0000 UTC m=+1732.918104561"
Nov 03 00:34:55.401687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:34:55.401633    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" podStartSLOduration=2.401621218 podStartE2EDuration="2.401621218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:34:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:34:55.399931782 +0000 UTC m=+1732.955283805" watchObservedRunningTime="2026-11-03 00:34:55.401621218 +0000 UTC m=+1732.956973242"
Nov 03 00:35:00.742947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.742914    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"]
Nov 03 00:35:00.747487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.747464    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.773936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.772437    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"]
Nov 03 00:35:00.775570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.775539    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-359f71-openshift-default-dockercfg-spn84\""
Nov 03 00:35:00.822631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822599    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.822840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822670    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-token\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.822840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822723    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-certs\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.822840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822788    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-envoy\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.822840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822818    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-credential-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.823092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822912    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-podinfo\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.823092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822937    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istiod-ca-cert\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.823092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.822994    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-data\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.823092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.823034    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftnmj\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-kube-api-access-ftnmj\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.926229 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.926185    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-podinfo\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.958118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.958087    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istiod-ca-cert\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.962461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.962430    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-data\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:00.968574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.968541    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftnmj\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-kube-api-access-ftnmj\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.000331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.979702    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.001372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.001340    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-token\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.002235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.002215    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-certs\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.002471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.002456    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-envoy\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.002569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.002557    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-credential-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.002848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.001898    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.003473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.957994    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-podinfo\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.003587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.959056    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istiod-ca-cert\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.003665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.003179    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-certs\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.003733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:00.968223    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-data\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.004027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.004008    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-credential-socket\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.033691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.020581    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-envoy\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.033691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.025793    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-token\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.033691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.032011    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftnmj\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-kube-api-access-ftnmj\") pod \"e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") " pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.077279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.070604    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:01.454936 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:01.454348    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1dba44_5c51_4c59_9cae_e122f75cd62b.slice/crio-4adb3e8fb2080dfa92ec5323f60df2a990373c081899f5b236426d4d4c6ed180 WatchSource:0}: Error finding container 4adb3e8fb2080dfa92ec5323f60df2a990373c081899f5b236426d4d4c6ed180: Status 404 returned error can't find the container with id 4adb3e8fb2080dfa92ec5323f60df2a990373c081899f5b236426d4d4c6ed180
Nov 03 00:35:01.454936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.454546    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"]
Nov 03 00:35:01.456945 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.456914    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:01.457150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.456979    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:01.457150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.457009    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:01.754339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:01.754239    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:35:02.410680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:02.410639    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" event={"ID":"7b1dba44-5c51-4c59-9cae-e122f75cd62b","Type":"ContainerStarted","Data":"1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9"}
Nov 03 00:35:02.410680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:02.410708    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" event={"ID":"7b1dba44-5c51-4c59-9cae-e122f75cd62b","Type":"ContainerStarted","Data":"4adb3e8fb2080dfa92ec5323f60df2a990373c081899f5b236426d4d4c6ed180"}
Nov 03 00:35:02.467315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:02.467216    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podStartSLOduration=2.467200599 podStartE2EDuration="2.467200599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:02.465298584 +0000 UTC m=+1740.020650620" watchObservedRunningTime="2026-11-03 00:35:02.467200599 +0000 UTC m=+1740.022552622"
Nov 03 00:35:03.075233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:03.074625    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:03.108282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:03.102625    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:03.625282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:03.621183    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:03.635607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:03.635569    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:35:04.293057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:04.292947    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.310281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:04.294915    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.567334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:04.531306    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.761061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:04.563373    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:04.761995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:04.616112    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.036283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.035418    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/health\": context deadline exceeded"
Nov 03 00:35:05.036283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.035857    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.300414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.300366    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.305283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.301753    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:35:05.313445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.313402    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.324284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.315806    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:35:05.781668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.774471    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:05.781668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.775915    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.781668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.776193    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": context deadline exceeded"
Nov 03 00:35:05.781668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.776761    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:35:05.781668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.776824    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:35:05.781668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.776864    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:35:05.860279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.844690    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"]
Nov 03 00:35:05.860279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:05.844968    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" containerID="cri-o://69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd" gracePeriod=30
Nov 03 00:35:06.062339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.036477    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.068427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.038484    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.069084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.069057    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:35:06.216394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.216350    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.358348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.330697    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.486419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.486374    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.709782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.709692    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 00:35:06.850321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.836098    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:06.860906 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.860859    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 00:35:06.969483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:06.889399    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 00:35:07.143279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:07.076324    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.407323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:07.406939    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.471469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:07.471438    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 00:35:07.499624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:07.498932    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.836476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:07.836356    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:07.872400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:07.872359    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.096475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.096390    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 00:35:08.122572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.119987    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.123675 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.123645    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 00:35:08.177421 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.177376    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:08.279836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.279803    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 00:35:08.672753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.664590    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 00:35:08.749294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.742044    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded"
Nov 03 00:35:08.977905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.977802    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 00:35:08.977905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:08.977872    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 00:35:09.741609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.741564    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"]
Nov 03 00:35:09.747822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.747797    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:09.765184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.765157    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"]
Nov 03 00:35:09.772962 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.772934    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-359f71\""
Nov 03 00:35:09.796168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.796123    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-359f71-dockercfg-g7ss9\""
Nov 03 00:35:09.810248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.810214    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"]
Nov 03 00:35:09.814479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.814460    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:09.836689 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.836656    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"]
Nov 03 00:35:09.896357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.893104    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct9pp\" (UniqueName: \"kubernetes.io/projected/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-kube-api-access-ct9pp\") pod \"payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:09.896670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.896641    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:09.900535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.900487    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dx24\" (UniqueName: \"kubernetes.io/projected/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-kube-api-access-7dx24\") pod \"payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:09.900684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:09.900618    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:10.020206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.009413    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct9pp\" (UniqueName: \"kubernetes.io/projected/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-kube-api-access-ct9pp\") pod \"payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:10.020206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.009478    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:10.051279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.043601    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:10.051279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.043804    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dx24\" (UniqueName: \"kubernetes.io/projected/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-kube-api-access-7dx24\") pod \"payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:10.051279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.043917    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:10.051279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.044941    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:10.061787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.061760    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct9pp\" (UniqueName: \"kubernetes.io/projected/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-kube-api-access-ct9pp\") pod \"payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:10.092324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.079967    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dx24\" (UniqueName: \"kubernetes.io/projected/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-kube-api-access-7dx24\") pod \"payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:10.128339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.128302    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:10.362750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.362710    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:10.500777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.500677    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"]
Nov 03 00:35:10.502126 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:10.502094    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff0a103_896a_43e3_a6b9_c02f8ffa23ff.slice/crio-fc3162907b090b285e037292c747ae4446810b56421f23734eeeae98612c0af4 WatchSource:0}: Error finding container fc3162907b090b285e037292c747ae4446810b56421f23734eeeae98612c0af4: Status 404 returned error can't find the container with id fc3162907b090b285e037292c747ae4446810b56421f23734eeeae98612c0af4
Nov 03 00:35:10.667381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.667353    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"]
Nov 03 00:35:10.667697 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:10.667674    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e57d6b5_7b1b_4f25_b7ce_edbe2f18a372.slice/crio-e797724e78988f2b473fb92affc3a8c1b4e005b280cbd1aed3ea7759bfb7c066 WatchSource:0}: Error finding container e797724e78988f2b473fb92affc3a8c1b4e005b280cbd1aed3ea7759bfb7c066: Status 404 returned error can't find the container with id e797724e78988f2b473fb92affc3a8c1b4e005b280cbd1aed3ea7759bfb7c066
Nov 03 00:35:10.757408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.757375    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" event={"ID":"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372","Type":"ContainerStarted","Data":"81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e"}
Nov 03 00:35:10.757805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.757441    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:10.757805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.757464    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" event={"ID":"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372","Type":"ContainerStarted","Data":"e797724e78988f2b473fb92affc3a8c1b4e005b280cbd1aed3ea7759bfb7c066"}
Nov 03 00:35:10.759396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.759371    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" event={"ID":"aff0a103-896a-43e3-a6b9-c02f8ffa23ff","Type":"ContainerStarted","Data":"55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6"}
Nov 03 00:35:10.759522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.759414    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" event={"ID":"aff0a103-896a-43e3-a6b9-c02f8ffa23ff","Type":"ContainerStarted","Data":"fc3162907b090b285e037292c747ae4446810b56421f23734eeeae98612c0af4"}
Nov 03 00:35:10.759522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.759435    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:10.796896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.796838    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" podStartSLOduration=1.7968228910000001 podStartE2EDuration="1.796822891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:10.794564842 +0000 UTC m=+1748.349916867" watchObservedRunningTime="2026-11-03 00:35:10.796822891 +0000 UTC m=+1748.352174911"
Nov 03 00:35:10.809688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.809650    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:35:10.809899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.809716    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:35:10.834799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:10.834742    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" podStartSLOduration=1.834727923 podStartE2EDuration="1.834727923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:10.834039566 +0000 UTC m=+1748.389391591" watchObservedRunningTime="2026-11-03 00:35:10.834727923 +0000 UTC m=+1748.390079946"
Nov 03 00:35:11.618995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.618968    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:35:11.670949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.670916    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-data\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.670966    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae8c127e-840d-4daa-afbb-c266e2cc5156-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c127e-840d-4daa-afbb-c266e2cc5156-istiod-ca-cert\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.671049    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-token\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.671093    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-certs\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.671121    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-credential-socket\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.671202    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-socket\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.671233    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-kube-api-access-4lrsx\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-kube-api-access-4lrsx\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.671346    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-podinfo\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.671621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.671461    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-envoy\") pod \"ae8c127e-840d-4daa-afbb-c266e2cc5156\" (UID: \"ae8c127e-840d-4daa-afbb-c266e2cc5156\") "
Nov 03 00:35:11.673008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.672601    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-data" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.673008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.672796    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-certs" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.673008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.672953    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-credential-socket" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.673008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.672983    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae8c127e-840d-4daa-afbb-c266e2cc5156-istiod-ca-cert" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:35:11.673008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.673006    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-socket" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.674004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.673968    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-token" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:11.675670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.675645    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-envoy" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:35:11.677230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.677195    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-kube-api-access-4lrsx" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "kube-api-access-4lrsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:35:11.677774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.677743    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-podinfo" pod "ae8c127e-840d-4daa-afbb-c266e2cc5156" (UID: "ae8c127e-840d-4daa-afbb-c266e2cc5156"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:35:11.767804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.767699    6000 generic.go:356] "Generic (PLEG): container finished" podID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerID="69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd" exitCode=0
Nov 03 00:35:11.767804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.767754    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"
Nov 03 00:35:11.767804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.767788    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" event={"ID":"ae8c127e-840d-4daa-afbb-c266e2cc5156","Type":"ContainerDied","Data":"69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd"}
Nov 03 00:35:11.768488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.767856    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf" event={"ID":"ae8c127e-840d-4daa-afbb-c266e2cc5156","Type":"ContainerDied","Data":"89a9a68c59a16699e68e45b5fcb79f64ee7344198e9d31ac5a9c43fa7c8ec527"}
Nov 03 00:35:11.768488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.767885    6000 scope.go:122] "RemoveContainer" containerID="69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd"
Nov 03 00:35:11.772776 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772757    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772779    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772788    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772795    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae8c127e-840d-4daa-afbb-c266e2cc5156-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772802    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772811    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772818    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772826    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae8c127e-840d-4daa-afbb-c266e2cc5156-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.772865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.772835    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrsx\" (UniqueName: \"kubernetes.io/projected/ae8c127e-840d-4daa-afbb-c266e2cc5156-kube-api-access-4lrsx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:35:11.778767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.778749    6000 scope.go:122] "RemoveContainer" containerID="69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd"
Nov 03 00:35:11.779063 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:35:11.779037    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd\": container with ID starting with 69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd not found: ID does not exist" containerID="69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd"
Nov 03 00:35:11.779161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.779072    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd"} err="failed to get container status \"69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd\": rpc error: code = NotFound desc = could not find container \"69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd\": container with ID starting with 69f88aefca6e67f7a40a8deb8e791add8bbfc532ac87c0a596f09179bfccd1dd not found: ID does not exist"
Nov 03 00:35:11.821111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.821082    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"]
Nov 03 00:35:11.825127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:11.825101    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-s9dbf"]
Nov 03 00:35:12.151468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.151437    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"]
Nov 03 00:35:12.152120 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.152105    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy"
Nov 03 00:35:12.152176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.152122    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy"
Nov 03 00:35:12.152233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.152224    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" containerName="istio-proxy"
Nov 03 00:35:12.156975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.156955    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.178008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.177978    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"]
Nov 03 00:35:12.178766 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.178239    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.178766 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.178603    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhskp\" (UniqueName: \"kubernetes.io/projected/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-kube-api-access-vhskp\") pod \"payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.252371 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.250310    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-e62935-dockercfg-nzlcd\""
Nov 03 00:35:12.268525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.268449    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-e62935\""
Nov 03 00:35:12.300990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.300841    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhskp\" (UniqueName: \"kubernetes.io/projected/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-kube-api-access-vhskp\") pod \"payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.312029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.300954    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.326600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.326550    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.359405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.358588    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhskp\" (UniqueName: \"kubernetes.io/projected/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-kube-api-access-vhskp\") pod \"payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.471215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.471133    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:12.572482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.572446    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae8c127e-840d-4daa-afbb-c266e2cc5156" path="/var/lib/kubelet/pods/ae8c127e-840d-4daa-afbb-c266e2cc5156/volumes"
Nov 03 00:35:12.788969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:12.788936    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"]
Nov 03 00:35:12.789352 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:12.789295    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1c0c4b2_8ae9_40cf_8efe_3d947241b77d.slice/crio-c80f58ceb9a40447cdd9667bb0d02420f95314daf5b9c4c810a7b1e02e11accc WatchSource:0}: Error finding container c80f58ceb9a40447cdd9667bb0d02420f95314daf5b9c4c810a7b1e02e11accc: Status 404 returned error can't find the container with id c80f58ceb9a40447cdd9667bb0d02420f95314daf5b9c4c810a7b1e02e11accc
Nov 03 00:35:13.785656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:13.785622    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" event={"ID":"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d","Type":"ContainerStarted","Data":"13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f"}
Nov 03 00:35:13.785852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:13.785677    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" event={"ID":"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d","Type":"ContainerStarted","Data":"c80f58ceb9a40447cdd9667bb0d02420f95314daf5b9c4c810a7b1e02e11accc"}
Nov 03 00:35:13.785852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:13.785698    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:13.825821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:13.825759    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" podStartSLOduration=1.8257456429999999 podStartE2EDuration="1.825745643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:13.822875721 +0000 UTC m=+1751.378227747" watchObservedRunningTime="2026-11-03 00:35:13.825745643 +0000 UTC m=+1751.381097666"
Nov 03 00:35:19.958037 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:35:19.957980    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:35:21.193471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:21.193418    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:21.318397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:21.318349    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:21.503569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:21.503439    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:35:21.831732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:21.831645    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 00:35:21.902053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:21.896237    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 00:35:22.060942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.060910    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"]
Nov 03 00:35:22.070297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.070207    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.084815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.084785    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"]
Nov 03 00:35:22.085981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.085956    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-359f71-serving-cert\""
Nov 03 00:35:22.147596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.147557    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b24645-1feb-4ed5-b735-cb817f512708-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.147891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.147818    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.148014 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.147918    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wczrj\" (UniqueName: \"kubernetes.io/projected/40b24645-1feb-4ed5-b735-cb817f512708-kube-api-access-wczrj\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.148014 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.147969    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.249422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.249388    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b24645-1feb-4ed5-b735-cb817f512708-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.249870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.249494    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.249870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.249546    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wczrj\" (UniqueName: \"kubernetes.io/projected/40b24645-1feb-4ed5-b735-cb817f512708-kube-api-access-wczrj\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.249870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.249575    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.250146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.250122    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b24645-1feb-4ed5-b735-cb817f512708-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.252139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.252117    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.252243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.252180    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.294315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.294279    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wczrj\" (UniqueName: \"kubernetes.io/projected/40b24645-1feb-4ed5-b735-cb817f512708-kube-api-access-wczrj\") pod \"maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.407402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.402677    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:22.753579 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:22.753550    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40b24645_1feb_4ed5_b735_cb817f512708.slice/crio-bc3005eaf707cb6d3509a700011419919550213cabf22993ea40f6cf8e8ac720 WatchSource:0}: Error finding container bc3005eaf707cb6d3509a700011419919550213cabf22993ea40f6cf8e8ac720: Status 404 returned error can't find the container with id bc3005eaf707cb6d3509a700011419919550213cabf22993ea40f6cf8e8ac720
Nov 03 00:35:22.753873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:22.753849    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"]
Nov 03 00:35:23.060283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:23.060171    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg" event={"ID":"40b24645-1feb-4ed5-b735-cb817f512708","Type":"ContainerStarted","Data":"bc3005eaf707cb6d3509a700011419919550213cabf22993ea40f6cf8e8ac720"}
Nov 03 00:35:24.019877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.019838    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"]
Nov 03 00:35:24.024178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.024157    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.039047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.039025    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-e62935-serving-cert\""
Nov 03 00:35:24.055838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.055806    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"]
Nov 03 00:35:24.070246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.069737    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg" event={"ID":"40b24645-1feb-4ed5-b735-cb817f512708","Type":"ContainerStarted","Data":"6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660"}
Nov 03 00:35:24.071345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.070809    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:24.084078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.084043    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.084245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.084168    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgphp\" (UniqueName: \"kubernetes.io/projected/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-kube-api-access-rgphp\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.084245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.084230    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.084400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.084271    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.120750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.120681    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg" podStartSLOduration=1.71056985 podStartE2EDuration="2.12066037s" totalImagesPullingTime="410.09052ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:24.109982419 +0000 UTC m=+1761.665334457" watchObservedRunningTime="2026-11-03 00:35:24.12066037 +0000 UTC m=+1761.676012397"
Nov 03 00:35:24.185600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.185568    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgphp\" (UniqueName: \"kubernetes.io/projected/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-kube-api-access-rgphp\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.185834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.185808    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.185947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.185866    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.186167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.186146    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.214974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.212059    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.216957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.216917    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.218066 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.218040    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.220575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.220544    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgphp\" (UniqueName: \"kubernetes.io/projected/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-kube-api-access-rgphp\") pod \"maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.341487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.341103    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:24.818420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.818355    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"]
Nov 03 00:35:24.818638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:24.818518    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 00:35:24.821732 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:24.821699    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd0ac0b_733e_4b26_ae0d_1ccacdaeeb36.slice/crio-1417e98c7b21ebf90561765f494d3f53c7b69e56d9a5327aa6c878354bdac9d6 WatchSource:0}: Error finding container 1417e98c7b21ebf90561765f494d3f53c7b69e56d9a5327aa6c878354bdac9d6: Status 404 returned error can't find the container with id 1417e98c7b21ebf90561765f494d3f53c7b69e56d9a5327aa6c878354bdac9d6
Nov 03 00:35:25.078682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:25.078589    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj" event={"ID":"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36","Type":"ContainerStarted","Data":"1417e98c7b21ebf90561765f494d3f53c7b69e56d9a5327aa6c878354bdac9d6"}
Nov 03 00:35:26.089467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:26.089406    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj" event={"ID":"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36","Type":"ContainerStarted","Data":"1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459"}
Nov 03 00:35:26.089980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:26.089500    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:26.130937 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:26.130217    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj" podStartSLOduration=2.696759528 podStartE2EDuration="3.130196963s" totalImagesPullingTime="433.437435ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:26.126658277 +0000 UTC m=+1763.682010302" watchObservedRunningTime="2026-11-03 00:35:26.130196963 +0000 UTC m=+1763.685548986"
Nov 03 00:35:30.085125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:30.085047    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 00:35:30.121655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:30.121624    6000 generic.go:356] "Generic (PLEG): container finished" podID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerID="352aba70745a9a63c4bd6b014a199fad79ec4cb612410abf2721e06f7ad85fd5" exitCode=0
Nov 03 00:35:30.121824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:30.121696    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" event={"ID":"0155e021-c45b-4fc5-a94e-a2df797f965f","Type":"ContainerDied","Data":"352aba70745a9a63c4bd6b014a199fad79ec4cb612410abf2721e06f7ad85fd5"}
Nov 03 00:35:30.122195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:30.122180    6000 scope.go:122] "RemoveContainer" containerID="352aba70745a9a63c4bd6b014a199fad79ec4cb612410abf2721e06f7ad85fd5"
Nov 03 00:35:30.124024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:30.123951    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:30.124171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:30.124032    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:30.124171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:30.124076    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:31.173071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:31.173034    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" event={"ID":"0155e021-c45b-4fc5-a94e-a2df797f965f","Type":"ContainerStarted","Data":"8e20ed78bfd07682ebe0fb77fb068e5b2b007261dc6a83651d51c38475add2ca"}
Nov 03 00:35:31.880594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:31.880560    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:35:31.881912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:31.881882    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 00:35:31.882041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:31.881950    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 00:35:32.121707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:32.121677    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 00:35:32.880503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:32.880473    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 00:35:32.880946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:32.880550    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 00:35:33.885279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:33.884729    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 00:35:33.885279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:33.884791    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 00:35:34.880544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:34.880512    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 00:35:34.880739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:34.880573    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 00:35:35.299613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.299082    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:35:35.515220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.515182    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"]
Nov 03 00:35:35.542161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.542130    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"]
Nov 03 00:35:35.542435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.542290    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:35.557430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.557403    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-9908e8\""
Nov 03 00:35:35.557739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.557704    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-9908e8-dockercfg-c6s79\""
Nov 03 00:35:35.835337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.771723    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"]
Nov 03 00:35:35.835337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.830653    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n874q\" (UniqueName: \"kubernetes.io/projected/de6eba37-c7cd-4239-a7fd-9b815b14dc79-kube-api-access-n874q\") pod \"payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:35.835337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.830738    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6eba37-c7cd-4239-a7fd-9b815b14dc79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:35.872473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.872440    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:35.902322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.887611    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused" start-of-body=
Nov 03 00:35:35.902322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.887664    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.146:15021/healthz/ready\": dial tcp 10.134.0.146:15021: connect: connection refused"
Nov 03 00:35:35.936574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.933372    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzssj\" (UniqueName: \"kubernetes.io/projected/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-kube-api-access-rzssj\") pod \"payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:35.936574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.933504    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n874q\" (UniqueName: \"kubernetes.io/projected/de6eba37-c7cd-4239-a7fd-9b815b14dc79-kube-api-access-n874q\") pod \"payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:35.936574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.933544    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:35.936574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.933606    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6eba37-c7cd-4239-a7fd-9b815b14dc79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:35.936574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.934570    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6eba37-c7cd-4239-a7fd-9b815b14dc79-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:35.936574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.934651    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"]
Nov 03 00:35:35.955048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:35.955025    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n874q\" (UniqueName: \"kubernetes.io/projected/de6eba37-c7cd-4239-a7fd-9b815b14dc79-kube-api-access-n874q\") pod \"payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:36.034678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.034643    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzssj\" (UniqueName: \"kubernetes.io/projected/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-kube-api-access-rzssj\") pod \"payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:36.034856 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.034705    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:36.035458 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.035440    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:36.053802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.053776    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzssj\" (UniqueName: \"kubernetes.io/projected/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-kube-api-access-rzssj\") pod \"payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:36.074883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.074841    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"]
Nov 03 00:35:36.078483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.078461    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.122272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.122227    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"]
Nov 03 00:35:36.136399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.136310    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.136780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.136754    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.137007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.136986    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.137142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.137125    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.137331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.137313    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.137468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.137452    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sp8xt\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-kube-api-access-sp8xt\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.137596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.137581    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.137711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.137695    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.137870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.137854    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.206281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.186440    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:36.239099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239055    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239172    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239219    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239287    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239318    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sp8xt\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-kube-api-access-sp8xt\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239362    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239385    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239476    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.239652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.239634    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.240127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.240099    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-data\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.240814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.240784    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istiod-ca-cert\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.241091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.241067    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-credential-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.241777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.241754    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-socket\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.242528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.242505    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-certs\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.245215 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.245188    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-podinfo\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.245989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.245945    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-envoy\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.249202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.249087    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:36.262098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.261020    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sp8xt\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-kube-api-access-sp8xt\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.262098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.262015    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-token\") pod \"e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") " pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.434708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.434613    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:36.555424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.555391    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"]
Nov 03 00:35:36.557546 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:36.557403    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde6eba37_c7cd_4239_a7fd_9b815b14dc79.slice/crio-98200b5dbee90990284130f49a5b84a7a7f25c121c15fe76999deaafcefe4d3e WatchSource:0}: Error finding container 98200b5dbee90990284130f49a5b84a7a7f25c121c15fe76999deaafcefe4d3e: Status 404 returned error can't find the container with id 98200b5dbee90990284130f49a5b84a7a7f25c121c15fe76999deaafcefe4d3e
Nov 03 00:35:36.621685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.621654    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"]
Nov 03 00:35:36.634009 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:36.626881    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f9048e_4dfe_4376_90e6_58c49ff9fc8c.slice/crio-968ccd6ff6399613abd96143f898262e06431c7e7da705e9458f6a51671a3d2a WatchSource:0}: Error finding container 968ccd6ff6399613abd96143f898262e06431c7e7da705e9458f6a51671a3d2a: Status 404 returned error can't find the container with id 968ccd6ff6399613abd96143f898262e06431c7e7da705e9458f6a51671a3d2a
Nov 03 00:35:36.759591 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:36.759555    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod009cbac3_f61f_4f63_9500_8ecf3a57afb0.slice/crio-66c00c46ddf28b14a68df35569ca4d47b1205329168d3f7a342f61a3875d69ce WatchSource:0}: Error finding container 66c00c46ddf28b14a68df35569ca4d47b1205329168d3f7a342f61a3875d69ce: Status 404 returned error can't find the container with id 66c00c46ddf28b14a68df35569ca4d47b1205329168d3f7a342f61a3875d69ce
Nov 03 00:35:36.760473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.760451    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"]
Nov 03 00:35:36.762761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.762728    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:36.762848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.762806    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:36.762848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.762840    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:36.885618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.885542    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:35:36.886769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:36.886746    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 00:35:37.357075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.357042    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" event={"ID":"68f9048e-4dfe-4376-90e6-58c49ff9fc8c","Type":"ContainerStarted","Data":"44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57"}
Nov 03 00:35:37.357237 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.357092    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" event={"ID":"68f9048e-4dfe-4376-90e6-58c49ff9fc8c","Type":"ContainerStarted","Data":"968ccd6ff6399613abd96143f898262e06431c7e7da705e9458f6a51671a3d2a"}
Nov 03 00:35:37.357343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.357244    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:37.358790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.358767    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" event={"ID":"de6eba37-c7cd-4239-a7fd-9b815b14dc79","Type":"ContainerStarted","Data":"b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0"}
Nov 03 00:35:37.358908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.358804    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" event={"ID":"de6eba37-c7cd-4239-a7fd-9b815b14dc79","Type":"ContainerStarted","Data":"98200b5dbee90990284130f49a5b84a7a7f25c121c15fe76999deaafcefe4d3e"}
Nov 03 00:35:37.358908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.358877    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:37.360691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.360660    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" event={"ID":"009cbac3-f61f-4f63-9500-8ecf3a57afb0","Type":"ContainerStarted","Data":"b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2"}
Nov 03 00:35:37.360846 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.360701    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" event={"ID":"009cbac3-f61f-4f63-9500-8ecf3a57afb0","Type":"ContainerStarted","Data":"66c00c46ddf28b14a68df35569ca4d47b1205329168d3f7a342f61a3875d69ce"}
Nov 03 00:35:37.412751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.412679    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" podStartSLOduration=2.412658828 podStartE2EDuration="2.412658828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:37.404240833 +0000 UTC m=+1774.959592859" watchObservedRunningTime="2026-11-03 00:35:37.412658828 +0000 UTC m=+1774.968010854"
Nov 03 00:35:37.435096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.435064    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:37.437001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.436965    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 00:35:37.437131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.437013    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 00:35:37.446501 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.446438    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podStartSLOduration=1.446314952 podStartE2EDuration="1.446314952s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:37.442591741 +0000 UTC m=+1774.997943789" watchObservedRunningTime="2026-11-03 00:35:37.446314952 +0000 UTC m=+1775.001666980"
Nov 03 00:35:37.481514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:37.481467    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" podStartSLOduration=2.481453411 podStartE2EDuration="2.481453411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:37.478988872 +0000 UTC m=+1775.034340896" watchObservedRunningTime="2026-11-03 00:35:37.481453411 +0000 UTC m=+1775.036805468"
Nov 03 00:35:38.435343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:38.435307    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 00:35:38.435816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:38.435375    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 00:35:39.435831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:39.435788    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 00:35:39.436353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:39.435848    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 00:35:40.435354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:40.435321    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 00:35:40.435518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:40.435376    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 00:35:41.435839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:41.435798    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 00:35:41.436242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:41.435856    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 00:35:42.435436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:42.435403    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 00:35:42.435635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:42.435457    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 00:35:43.446697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:43.446660    6000 patch_prober.go:29] interesting pod/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body=
Nov 03 00:35:43.447342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:43.447308    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused"
Nov 03 00:35:44.668327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:44.668295    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:44.668810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:44.668744    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:44.669653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:44.669629    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:35:45.086351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:45.086249    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:35:48.510608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:48.505397    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 00:35:48.510608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:48.508425    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 00:35:49.918340 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:35:49.918301    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:35:53.509229 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.509196    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"]
Nov 03 00:35:53.513072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.513048    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:53.532079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.532056    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-2858fb\""
Nov 03 00:35:53.534324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.534295    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"]
Nov 03 00:35:53.578480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.578448    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"]
Nov 03 00:35:53.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.581911    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.600758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.600728    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-2858fb-dockercfg-ghkxs\""
Nov 03 00:35:53.602895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.602875    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"]
Nov 03 00:35:53.743832 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.743790    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d7d40-68cf-482f-86fa-34f5af94813c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:53.744071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.744048    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qpj\" (UniqueName: \"kubernetes.io/projected/2f71ee87-6e9f-4e44-b400-577011399180-kube-api-access-w7qpj\") pod \"payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.744337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.744312    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cs2x\" (UniqueName: \"kubernetes.io/projected/d17d7d40-68cf-482f-86fa-34f5af94813c-kube-api-access-2cs2x\") pod \"payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:53.744707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.744688    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f71ee87-6e9f-4e44-b400-577011399180-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.859277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.856298    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cs2x\" (UniqueName: \"kubernetes.io/projected/d17d7d40-68cf-482f-86fa-34f5af94813c-kube-api-access-2cs2x\") pod \"payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:53.859277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.856485    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f71ee87-6e9f-4e44-b400-577011399180-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.859277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.856595    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d7d40-68cf-482f-86fa-34f5af94813c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:53.859277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.856626    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qpj\" (UniqueName: \"kubernetes.io/projected/2f71ee87-6e9f-4e44-b400-577011399180-kube-api-access-w7qpj\") pod \"payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.859277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.858109    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f71ee87-6e9f-4e44-b400-577011399180-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.859277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.858873    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d7d40-68cf-482f-86fa-34f5af94813c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:53.898765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.894123    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qpj\" (UniqueName: \"kubernetes.io/projected/2f71ee87-6e9f-4e44-b400-577011399180-kube-api-access-w7qpj\") pod \"payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.913054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.907593    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:53.914040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:53.913640    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cs2x\" (UniqueName: \"kubernetes.io/projected/d17d7d40-68cf-482f-86fa-34f5af94813c-kube-api-access-2cs2x\") pod \"payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:54.129221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:54.129139    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:54.233412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:54.232709    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"]
Nov 03 00:35:54.245347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:54.245287    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" event={"ID":"2f71ee87-6e9f-4e44-b400-577011399180","Type":"ContainerStarted","Data":"16adf825e05e3bf4a1a753cdfb52107ec9f0dbc1e647b8ec0f753ed6209d007a"}
Nov 03 00:35:54.444605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:54.444443    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"]
Nov 03 00:35:54.447677 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:54.447646    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd17d7d40_68cf_482f_86fa_34f5af94813c.slice/crio-a0d160e99e437942ef0151047ef122fbcf3cbb3e35a4e98efe0234ccfb8fa6a4 WatchSource:0}: Error finding container a0d160e99e437942ef0151047ef122fbcf3cbb3e35a4e98efe0234ccfb8fa6a4: Status 404 returned error can't find the container with id a0d160e99e437942ef0151047ef122fbcf3cbb3e35a4e98efe0234ccfb8fa6a4
Nov 03 00:35:55.254798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:55.254760    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" event={"ID":"2f71ee87-6e9f-4e44-b400-577011399180","Type":"ContainerStarted","Data":"3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a"}
Nov 03 00:35:55.255472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:55.255011    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:35:55.256765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:55.256739    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" event={"ID":"d17d7d40-68cf-482f-86fa-34f5af94813c","Type":"ContainerStarted","Data":"72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28"}
Nov 03 00:35:55.256889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:55.256781    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" event={"ID":"d17d7d40-68cf-482f-86fa-34f5af94813c","Type":"ContainerStarted","Data":"a0d160e99e437942ef0151047ef122fbcf3cbb3e35a4e98efe0234ccfb8fa6a4"}
Nov 03 00:35:55.257400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:55.257381    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:35:55.293206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:55.293152    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" podStartSLOduration=2.293136617 podStartE2EDuration="2.293136617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:55.291034402 +0000 UTC m=+1792.846386439" watchObservedRunningTime="2026-11-03 00:35:55.293136617 +0000 UTC m=+1792.848488641"
Nov 03 00:35:55.327245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:55.327178    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" podStartSLOduration=2.327157793 podStartE2EDuration="2.327157793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:55.326628806 +0000 UTC m=+1792.881980830" watchObservedRunningTime="2026-11-03 00:35:55.327157793 +0000 UTC m=+1792.882509828"
Nov 03 00:35:57.806154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.806116    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"]
Nov 03 00:35:57.809390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.809373    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:57.835102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.835063    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-a07a90\""
Nov 03 00:35:57.835355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.835173    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-a07a90-dockercfg-sjg5k\""
Nov 03 00:35:57.845447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.845400    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"]
Nov 03 00:35:57.846462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.846435    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"]
Nov 03 00:35:57.851140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.851101    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:57.867917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.867889    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"]
Nov 03 00:35:57.874141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.874104    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvxdw\" (UniqueName: \"kubernetes.io/projected/21af1f82-17cb-412b-856e-9f2592be3b4e-kube-api-access-bvxdw\") pod \"payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:57.874337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.874217    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee39d2b9-9e10-4947-b669-300e04a03415-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:57.874337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.874326    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hkqx\" (UniqueName: \"kubernetes.io/projected/ee39d2b9-9e10-4947-b669-300e04a03415-kube-api-access-8hkqx\") pod \"payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:57.874478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.874366    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af1f82-17cb-412b-856e-9f2592be3b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:57.999281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:57.984414    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvxdw\" (UniqueName: \"kubernetes.io/projected/21af1f82-17cb-412b-856e-9f2592be3b4e-kube-api-access-bvxdw\") pod \"payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:58.036398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.036355    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee39d2b9-9e10-4947-b669-300e04a03415-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:58.036711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.036673    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hkqx\" (UniqueName: \"kubernetes.io/projected/ee39d2b9-9e10-4947-b669-300e04a03415-kube-api-access-8hkqx\") pod \"payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:58.036864 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.036850    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af1f82-17cb-412b-856e-9f2592be3b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:58.070336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.038067    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af1f82-17cb-412b-856e-9f2592be3b4e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:58.091282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.060180    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee39d2b9-9e10-4947-b669-300e04a03415-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:58.156274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.156224    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvxdw\" (UniqueName: \"kubernetes.io/projected/21af1f82-17cb-412b-856e-9f2592be3b4e-kube-api-access-bvxdw\") pod \"payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:58.157126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.156905    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hkqx\" (UniqueName: \"kubernetes.io/projected/ee39d2b9-9e10-4947-b669-300e04a03415-kube-api-access-8hkqx\") pod \"payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:58.175998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.175963    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:58.289197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.288651    6000 generic.go:356] "Generic (PLEG): container finished" podID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerID="1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9" exitCode=0
Nov 03 00:35:58.289197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.288728    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" event={"ID":"7b1dba44-5c51-4c59-9cae-e122f75cd62b","Type":"ContainerDied","Data":"1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9"}
Nov 03 00:35:58.289479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.289428    6000 scope.go:122] "RemoveContainer" containerID="1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9"
Nov 03 00:35:58.292826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.292005    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:58.292826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.292095    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:58.292826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.292140    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:35:58.424917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.424887    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:58.543973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.541144    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"]
Nov 03 00:35:58.553807 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:35:58.552483    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21af1f82_17cb_412b_856e_9f2592be3b4e.slice/crio-1e4171c01778ef564b047aaca16afefb5fdbed116f74ebe422147a721bde6c78 WatchSource:0}: Error finding container 1e4171c01778ef564b047aaca16afefb5fdbed116f74ebe422147a721bde6c78: Status 404 returned error can't find the container with id 1e4171c01778ef564b047aaca16afefb5fdbed116f74ebe422147a721bde6c78
Nov 03 00:35:58.923280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:58.888230    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"]
Nov 03 00:35:59.297249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.297152    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" event={"ID":"7b1dba44-5c51-4c59-9cae-e122f75cd62b","Type":"ContainerStarted","Data":"6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782"}
Nov 03 00:35:59.298956 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.298931    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" event={"ID":"ee39d2b9-9e10-4947-b669-300e04a03415","Type":"ContainerStarted","Data":"50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca"}
Nov 03 00:35:59.299532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.298981    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" event={"ID":"ee39d2b9-9e10-4947-b669-300e04a03415","Type":"ContainerStarted","Data":"c54f6ed7af2889bf033a7007a989adfe80864657619782c4425c9336c0cb87d4"}
Nov 03 00:35:59.299532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.299013    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:35:59.300509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.300488    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" event={"ID":"21af1f82-17cb-412b-856e-9f2592be3b4e","Type":"ContainerStarted","Data":"9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20"}
Nov 03 00:35:59.300637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.300525    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" event={"ID":"21af1f82-17cb-412b-856e-9f2592be3b4e","Type":"ContainerStarted","Data":"1e4171c01778ef564b047aaca16afefb5fdbed116f74ebe422147a721bde6c78"}
Nov 03 00:35:59.300700 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.300638    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:35:59.375857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.375811    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" podStartSLOduration=2.375796662 podStartE2EDuration="2.375796662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:59.372680998 +0000 UTC m=+1796.928033035" watchObservedRunningTime="2026-11-03 00:35:59.375796662 +0000 UTC m=+1796.931148685"
Nov 03 00:35:59.410230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:35:59.410183    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" podStartSLOduration=2.410170325 podStartE2EDuration="2.410170325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:35:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:35:59.407827017 +0000 UTC m=+1796.963179041" watchObservedRunningTime="2026-11-03 00:35:59.410170325 +0000 UTC m=+1796.965522348"
Nov 03 00:36:00.070927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:00.070876    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:36:00.072040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:00.072014    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 00:36:00.072165 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:00.072072    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 00:36:01.071449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:01.071414    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 00:36:01.071856 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:01.071469    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 00:36:02.070978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:02.070941    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 00:36:02.071149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:02.071001    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 00:36:03.071308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:03.071248    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 00:36:03.071718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:03.071329    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 00:36:03.621910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:03.621248    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:36:04.077335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:04.072687    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 00:36:04.077335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:04.072752    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 00:36:05.075281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:05.071687    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused" start-of-body=
Nov 03 00:36:05.075281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:05.071755    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.149:15021/healthz/ready\": dial tcp 10.134.0.149:15021: connect: connection refused"
Nov 03 00:36:06.076731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:06.076693    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:36:06.077912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:06.077890    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:36:06.268293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:06.268237    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 00:36:06.268487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:06.268319    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 00:36:10.321074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:10.321037    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 00:36:10.321567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:10.321327    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 00:36:13.009883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.009798    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"]
Nov 03 00:36:13.020650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.020614    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.034323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.034289    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"]
Nov 03 00:36:13.041050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.041016    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-2858fb-serving-cert\""
Nov 03 00:36:13.111666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.111633    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e4c79f-9376-4904-9d4e-6fbb7208413a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.111857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.111839    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hlw4\" (UniqueName: \"kubernetes.io/projected/80e4c79f-9376-4904-9d4e-6fbb7208413a-kube-api-access-9hlw4\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.111908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.111868    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.111944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.111928    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.213815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.213776    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hlw4\" (UniqueName: \"kubernetes.io/projected/80e4c79f-9376-4904-9d4e-6fbb7208413a-kube-api-access-9hlw4\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.213815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.213819    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.214094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.213877    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.214094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.214069    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e4c79f-9376-4904-9d4e-6fbb7208413a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.214798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.214772    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e4c79f-9376-4904-9d4e-6fbb7208413a-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.216669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.216644    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.216819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.216797    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.232405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.232380    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hlw4\" (UniqueName: \"kubernetes.io/projected/80e4c79f-9376-4904-9d4e-6fbb7208413a-kube-api-access-9hlw4\") pod \"maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.347393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.345862    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:13.672176 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:36:13.672148    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80e4c79f_9376_4904_9d4e_6fbb7208413a.slice/crio-2445b9b498961c37ef4ab4d24ca04f07cca27735176debaa49aa0e765421697d WatchSource:0}: Error finding container 2445b9b498961c37ef4ab4d24ca04f07cca27735176debaa49aa0e765421697d: Status 404 returned error can't find the container with id 2445b9b498961c37ef4ab4d24ca04f07cca27735176debaa49aa0e765421697d
Nov 03 00:36:13.672609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:13.672586    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"]
Nov 03 00:36:14.471201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:14.471152    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw" event={"ID":"80e4c79f-9376-4904-9d4e-6fbb7208413a","Type":"ContainerStarted","Data":"d803e8c16c7fa90244ab6d52a58bddd0e27295a6566d90c7eb568c131dfeb4ce"}
Nov 03 00:36:14.471642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:14.471226    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw" event={"ID":"80e4c79f-9376-4904-9d4e-6fbb7208413a","Type":"ContainerStarted","Data":"2445b9b498961c37ef4ab4d24ca04f07cca27735176debaa49aa0e765421697d"}
Nov 03 00:36:14.471642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:14.471274    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:14.512323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:14.512248    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw" podStartSLOduration=2.155697566 podStartE2EDuration="2.512233416s" totalImagesPullingTime="356.53585ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:14.508806038 +0000 UTC m=+1812.064158061" watchObservedRunningTime="2026-11-03 00:36:14.512233416 +0000 UTC m=+1812.067585438"
Nov 03 00:36:18.786556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.786520    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"]
Nov 03 00:36:18.790548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.790525    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.820582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.818967    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c59124\"/\"default-dockercfg-6gmvx\""
Nov 03 00:36:18.823415 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.823392    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"]
Nov 03 00:36:18.823551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.823539    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c59124\"/\"openshift-service-ca.crt\""
Nov 03 00:36:18.823636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.823626    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-c59124\"/\"autores-model-c59124-kserve-self-signed-certs\""
Nov 03 00:36:18.823701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.823692    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-c59124\"/\"kube-root-ca.crt\""
Nov 03 00:36:18.853142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.853108    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-model-cache\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.853350 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.853154    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-dshm\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.853350 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.853280    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc8tt\" (UniqueName: \"kubernetes.io/projected/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-kube-api-access-zc8tt\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.853494 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.853473    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-home\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.853556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.853497    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tmp-dir\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.853556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.853516    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tls-certs\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.954432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.954399    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-model-cache\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.954606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.954447    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-dshm\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.954606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.954498    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc8tt\" (UniqueName: \"kubernetes.io/projected/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-kube-api-access-zc8tt\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.954757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.954738    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-home\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.954803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.954781    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tmp-dir\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.954852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.954791    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-model-cache\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.954852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.954806    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tls-certs\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.955035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.955016    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-home\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.955097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.955069    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tmp-dir\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.956996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.956970    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-dshm\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.957288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.957235    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tls-certs\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:18.973211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:18.973178    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc8tt\" (UniqueName: \"kubernetes.io/projected/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-kube-api-access-zc8tt\") pod \"autores-model-c59124-kserve-7c897844df-mkttg\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") " pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:19.070138 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:19.070039    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:36:19.114279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:19.113553    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:19.762199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:19.762167    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"]
Nov 03 00:36:19.898602 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:36:19.898565    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:36:20.492210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:20.492178    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 00:36:20.535625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:20.535586    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg" event={"ID":"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4","Type":"ContainerStarted","Data":"75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad"}
Nov 03 00:36:20.535785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:20.535633    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg" event={"ID":"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4","Type":"ContainerStarted","Data":"e0469ccd1f444960b6146b6897751839fc09dc86dcdb6613f1e9994c51f6b0fc"}
Nov 03 00:36:20.581632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:20.581579    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg" podStartSLOduration=2.581562563 podStartE2EDuration="2.581562563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:20.578651781 +0000 UTC m=+1818.134003805" watchObservedRunningTime="2026-11-03 00:36:20.581562563 +0000 UTC m=+1818.136914586"
Nov 03 00:36:21.114433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:21.114402    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:29.115017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:29.114979    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:29.128056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:29.128028    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:29.607607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:29.607579    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:36:36.434951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.434908    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"]
Nov 03 00:36:36.439554 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.439528    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:36.460195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.460168    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"]
Nov 03 00:36:36.467807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.467778    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-20109d-dockercfg-vbg4h\""
Nov 03 00:36:36.467965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.467820    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-20109d\""
Nov 03 00:36:36.597333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.592635    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:36.597333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.592831    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2n9gm\" (UniqueName: \"kubernetes.io/projected/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-kube-api-access-2n9gm\") pod \"payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:36.671869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.671828    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"]
Nov 03 00:36:36.697688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.695825    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:36.697688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.695976    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2n9gm\" (UniqueName: \"kubernetes.io/projected/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-kube-api-access-2n9gm\") pod \"payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:36.697688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.697198    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:36.721731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.721703    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:36.722096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.721951    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"]
Nov 03 00:36:36.805280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.790480    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2n9gm\" (UniqueName: \"kubernetes.io/projected/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-kube-api-access-2n9gm\") pod \"payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:36.943170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.943123    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:36.943760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:36.943732    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs6hz\" (UniqueName: \"kubernetes.io/projected/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-kube-api-access-bs6hz\") pod \"payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:37.067119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.065238    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:37.067119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.065483    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs6hz\" (UniqueName: \"kubernetes.io/projected/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-kube-api-access-bs6hz\") pod \"payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:37.067119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.067023    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:37.068469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.068444    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:37.101936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.095335    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs6hz\" (UniqueName: \"kubernetes.io/projected/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-kube-api-access-bs6hz\") pod \"payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:37.351766 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.351459    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:37.508228 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:36:37.507641    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3adb0154_70eb_464b_bfcb_3ffc3d5476e0.slice/crio-f684623fbfe6a64bda4f4763d8a4cb440ab415bd285033d2a66dfe8c37964904 WatchSource:0}: Error finding container f684623fbfe6a64bda4f4763d8a4cb440ab415bd285033d2a66dfe8c37964904: Status 404 returned error can't find the container with id f684623fbfe6a64bda4f4763d8a4cb440ab415bd285033d2a66dfe8c37964904
Nov 03 00:36:37.509367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.509336    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"]
Nov 03 00:36:37.669719 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:36:37.669687    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2dde3ef_cadd_4c34_a7ef_ca4b27817ff6.slice/crio-8f35ee88c56e8c8bd01a801889fda45cb175c82d9a2bde717a8f17c313727c72 WatchSource:0}: Error finding container 8f35ee88c56e8c8bd01a801889fda45cb175c82d9a2bde717a8f17c313727c72: Status 404 returned error can't find the container with id 8f35ee88c56e8c8bd01a801889fda45cb175c82d9a2bde717a8f17c313727c72
Nov 03 00:36:37.669914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.669891    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"]
Nov 03 00:36:37.728461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.728432    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" event={"ID":"3adb0154-70eb-464b-bfcb-3ffc3d5476e0","Type":"ContainerStarted","Data":"18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914"}
Nov 03 00:36:37.728624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.728490    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" event={"ID":"3adb0154-70eb-464b-bfcb-3ffc3d5476e0","Type":"ContainerStarted","Data":"f684623fbfe6a64bda4f4763d8a4cb440ab415bd285033d2a66dfe8c37964904"}
Nov 03 00:36:37.728624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.728522    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:37.729793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.729769    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" event={"ID":"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6","Type":"ContainerStarted","Data":"8f35ee88c56e8c8bd01a801889fda45cb175c82d9a2bde717a8f17c313727c72"}
Nov 03 00:36:37.768537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:37.768489    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" podStartSLOduration=1.768469048 podStartE2EDuration="1.768469048s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:37.767326215 +0000 UTC m=+1835.322678266" watchObservedRunningTime="2026-11-03 00:36:37.768469048 +0000 UTC m=+1835.323821072"
Nov 03 00:36:38.737763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:38.737723    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" event={"ID":"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6","Type":"ContainerStarted","Data":"cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37"}
Nov 03 00:36:38.738285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:38.737892    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:38.776383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:38.776332    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" podStartSLOduration=2.776316935 podStartE2EDuration="2.776316935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:38.773688061 +0000 UTC m=+1836.329040086" watchObservedRunningTime="2026-11-03 00:36:38.776316935 +0000 UTC m=+1836.331668971"
Nov 03 00:36:48.741413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:48.741386    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 00:36:49.747798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:49.747773    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 00:36:49.909143 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:36:49.908412    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:36:51.008287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.007716    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"]
Nov 03 00:36:51.019306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.015892    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.046693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.046662    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-c59124-openshift-default-dockercfg-7f78f\""
Nov 03 00:36:51.052203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.052173    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"]
Nov 03 00:36:51.077061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077031    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istiod-ca-cert\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077089    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-data\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077202    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-podinfo\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077468    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-token\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077603    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077654    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-credential-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077794    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-envoy\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077871    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-certs\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.077946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.077928    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p8vm\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-kube-api-access-5p8vm\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.178981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.178941    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-envoy\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179016    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-certs\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179060    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p8vm\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-kube-api-access-5p8vm\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179118    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istiod-ca-cert\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179150    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-data\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179170    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-podinfo\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179287    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-token\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179379    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179406    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-credential-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179532    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-certs\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179884    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-data\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179904    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.179993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.179931    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istiod-ca-cert\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.180115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.180096    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-credential-socket\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.181623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.181598    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-envoy\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.182180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.182159    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-podinfo\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.199083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.199064    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-token\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.199277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.199240    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p8vm\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-kube-api-access-5p8vm\") pod \"e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") " pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.338481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.338397    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:51.464271 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.464224    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:36:51.651977 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:36:51.651942    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412aff7a_a67a_4e5c_8e4e_fcb1dadea987.slice/crio-7cdf73962872ad8c0a90a8ccdc8d76151ac4527971e9741d017e084fdec4d209 WatchSource:0}: Error finding container 7cdf73962872ad8c0a90a8ccdc8d76151ac4527971e9741d017e084fdec4d209: Status 404 returned error can't find the container with id 7cdf73962872ad8c0a90a8ccdc8d76151ac4527971e9741d017e084fdec4d209
Nov 03 00:36:51.652116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.652048    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"]
Nov 03 00:36:51.654855 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.654822    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:36:51.654960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.654907    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:36:51.654960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.654948    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:36:51.934076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.933977    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" event={"ID":"412aff7a-a67a-4e5c-8e4e-fcb1dadea987","Type":"ContainerStarted","Data":"8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b"}
Nov 03 00:36:51.934076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.934032    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" event={"ID":"412aff7a-a67a-4e5c-8e4e-fcb1dadea987","Type":"ContainerStarted","Data":"7cdf73962872ad8c0a90a8ccdc8d76151ac4527971e9741d017e084fdec4d209"}
Nov 03 00:36:51.977482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:51.977427    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podStartSLOduration=1.977411818 podStartE2EDuration="1.977411818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:36:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:36:51.973148567 +0000 UTC m=+1849.528500613" watchObservedRunningTime="2026-11-03 00:36:51.977411818 +0000 UTC m=+1849.532763841"
Nov 03 00:36:52.339546 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:52.339461    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:52.341591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:52.341563    6000 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 00:36:52.341724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:52.341615    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 00:36:52.958187 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:52.958149    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"]
Nov 03 00:36:53.338897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:53.338805    6000 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 00:36:53.338897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:53.338864    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 00:36:54.339631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:54.339591    6000 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 00:36:54.340060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:54.339659    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 00:36:55.342639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:55.342601    6000 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 00:36:55.343039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:55.342685    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 00:36:56.339003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:56.338964    6000 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 00:36:56.339218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:56.339022    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 00:36:57.339132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:57.339095    6000 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body=
Nov 03 00:36:57.339672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:57.339155    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused"
Nov 03 00:36:58.343525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:58.343495    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:58.343949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:58.343938    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:36:58.344848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:36:58.344827    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:37:00.819126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.819086    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"]
Nov 03 00:37:00.824018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.823995    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.841747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.841714    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"]
Nov 03 00:37:00.852607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.849758    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-20109d-serving-cert\""
Nov 03 00:37:00.887276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.887222    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.887462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.887445    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/619a36e0-3a99-47b1-ae3a-0a9395e118a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.887583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.887557    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.887716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.887692    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/619a36e0-3a99-47b1-ae3a-0a9395e118a4-kube-api-access-rmjcd\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.989487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.989453    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/619a36e0-3a99-47b1-ae3a-0a9395e118a4-kube-api-access-rmjcd\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.989652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.989569    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.989719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.989659    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/619a36e0-3a99-47b1-ae3a-0a9395e118a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.989719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.989704    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.990365 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.990336    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/619a36e0-3a99-47b1-ae3a-0a9395e118a4-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.992353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.992331    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:00.992471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:00.992413    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:01.008321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:01.008294    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/619a36e0-3a99-47b1-ae3a-0a9395e118a4-kube-api-access-rmjcd\") pod \"maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:01.144286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:01.142490    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:01.477489 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:37:01.477460    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619a36e0_3a99_47b1_ae3a_0a9395e118a4.slice/crio-c28baeddbdb14503069201529ea138768b388e48f70c71e26cfbe3613fc2e18f WatchSource:0}: Error finding container c28baeddbdb14503069201529ea138768b388e48f70c71e26cfbe3613fc2e18f: Status 404 returned error can't find the container with id c28baeddbdb14503069201529ea138768b388e48f70c71e26cfbe3613fc2e18f
Nov 03 00:37:01.478791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:01.478765    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"]
Nov 03 00:37:02.052289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:02.051041    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r" event={"ID":"619a36e0-3a99-47b1-ae3a-0a9395e118a4","Type":"ContainerStarted","Data":"30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706"}
Nov 03 00:37:02.052289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:02.051116    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r" event={"ID":"619a36e0-3a99-47b1-ae3a-0a9395e118a4","Type":"ContainerStarted","Data":"c28baeddbdb14503069201529ea138768b388e48f70c71e26cfbe3613fc2e18f"}
Nov 03 00:37:02.061517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:02.052323    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:02.122281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:02.119147    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r" podStartSLOduration=1.764522598 podStartE2EDuration="2.119127655s" totalImagesPullingTime="354.605057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:02.115773659 +0000 UTC m=+1859.671125683" watchObservedRunningTime="2026-11-03 00:37:02.119127655 +0000 UTC m=+1859.674479679"
Nov 03 00:37:07.985721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:07.985676    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg" podUID="78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" containerName="main" containerID="cri-o://75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad" gracePeriod=45
Nov 03 00:37:08.331196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.331164    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:37:08.405136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405098    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tmp-dir\") pod \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") "
Nov 03 00:37:08.405323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405194    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-dshm\") pod \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") "
Nov 03 00:37:08.405397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405369    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-home\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-home\") pod \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") "
Nov 03 00:37:08.405456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405440    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-kube-api-access-zc8tt\" (UniqueName: \"kubernetes.io/projected/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-kube-api-access-zc8tt\") pod \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") "
Nov 03 00:37:08.405502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405472    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tmp-dir" pod "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" (UID: "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:37:08.405599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405582    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-model-cache\") pod \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") "
Nov 03 00:37:08.405675 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405618    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tls-certs\") pod \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\" (UID: \"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4\") "
Nov 03 00:37:08.405740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405678    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-home" pod "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" (UID: "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:37:08.405893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.405864    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-model-cache" pod "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" (UID: "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:37:08.406360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.406337    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:37:08.406510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.406487    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:37:08.406510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.406513    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:37:08.408213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.408184    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-kube-api-access-zc8tt" pod "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" (UID: "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4"). InnerVolumeSpecName "kube-api-access-zc8tt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:37:08.408471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.408450    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-dshm" pod "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" (UID: "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:37:08.408626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.408607    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tls-certs" pod "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" (UID: "78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:37:08.508449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.508411    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:37:08.508449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.508444    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:37:08.508681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:08.508458    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zc8tt\" (UniqueName: \"kubernetes.io/projected/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4-kube-api-access-zc8tt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:37:09.071188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.071161    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 00:37:09.106322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.106244    6000 generic.go:356] "Generic (PLEG): container finished" podID="78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" containerID="75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad" exitCode=0
Nov 03 00:37:09.106322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.106301    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg" event={"ID":"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4","Type":"ContainerDied","Data":"75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad"}
Nov 03 00:37:09.106551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.106345    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg" event={"ID":"78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4","Type":"ContainerDied","Data":"e0469ccd1f444960b6146b6897751839fc09dc86dcdb6613f1e9994c51f6b0fc"}
Nov 03 00:37:09.106551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.106365    6000 scope.go:122] "RemoveContainer" containerID="75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad"
Nov 03 00:37:09.106551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.106307    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"
Nov 03 00:37:09.119016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.118985    6000 scope.go:122] "RemoveContainer" containerID="75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad"
Nov 03 00:37:09.119453 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:37:09.119421    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad\": container with ID starting with 75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad not found: ID does not exist" containerID="75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad"
Nov 03 00:37:09.119584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.119461    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad"} err="failed to get container status \"75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad\": rpc error: code = NotFound desc = could not find container \"75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad\": container with ID starting with 75a7d900af471e76abb1cf33ab89cef91a2ce92552f067f2235c6892301500ad not found: ID does not exist"
Nov 03 00:37:09.160038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.160005    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"]
Nov 03 00:37:09.165334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:09.165304    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-c59124/autores-model-c59124-kserve-7c897844df-mkttg"]
Nov 03 00:37:10.569412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:10.569378    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" path="/var/lib/kubelet/pods/78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4/volumes"
Nov 03 00:37:18.647828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.647611    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"]
Nov 03 00:37:18.648552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.648531    6000 cpu_manager.go:414] "containerMap: removing container" podUID="78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" containerName="main"
Nov 03 00:37:18.648664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.648556    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" containerName="main"
Nov 03 00:37:18.648787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.648773    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="78ad5bc6-1b24-499a-b8bc-c3eacdf71ed4" containerName="main"
Nov 03 00:37:18.654951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.654922    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.671985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.671956    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"]
Nov 03 00:37:18.679319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.678683    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"kube-root-ca.crt\""
Nov 03 00:37:18.679319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.678853    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"openshift-service-ca.crt\""
Nov 03 00:37:18.681919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.680865    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"default-dockercfg-s2tsc\""
Nov 03 00:37:18.681919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.680865    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-7ef1b6\"/\"auth-test-model-7ef1b6-kserve-self-signed-certs\""
Nov 03 00:37:18.775524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.775496    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81963819-4fe1-4e23-8c88-908fa1e21a62-tls-certs\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.775710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.775548    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p78nt\" (UniqueName: \"kubernetes.io/projected/81963819-4fe1-4e23-8c88-908fa1e21a62-kube-api-access-p78nt\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.775710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.775657    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-dshm\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.775806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.775765    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-model-cache\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.775806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.775790    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-tmp-dir\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.775895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.775835    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-home\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.877974    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-home\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.878156    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81963819-4fe1-4e23-8c88-908fa1e21a62-tls-certs\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.878222    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p78nt\" (UniqueName: \"kubernetes.io/projected/81963819-4fe1-4e23-8c88-908fa1e21a62-kube-api-access-p78nt\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.878347    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-dshm\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.878484    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-model-cache\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.878523    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-tmp-dir\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.879132    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-tmp-dir\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.879475    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-home\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.892302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.880553    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-model-cache\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.932380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.923339    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p78nt\" (UniqueName: \"kubernetes.io/projected/81963819-4fe1-4e23-8c88-908fa1e21a62-kube-api-access-p78nt\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.933129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.933089    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-dshm\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.965660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.965629    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81963819-4fe1-4e23-8c88-908fa1e21a62-tls-certs\") pod \"auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") " pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:18.976353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:18.973306    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:19.357172 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:37:19.357141    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81963819_4fe1_4e23_8c88_908fa1e21a62.slice/crio-eb5c82d6b3dba172c5bf0950a1fd3e27b5ffc1c21a79000075aac8fc899b707d WatchSource:0}: Error finding container eb5c82d6b3dba172c5bf0950a1fd3e27b5ffc1c21a79000075aac8fc899b707d: Status 404 returned error can't find the container with id eb5c82d6b3dba172c5bf0950a1fd3e27b5ffc1c21a79000075aac8fc899b707d
Nov 03 00:37:19.358200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:19.358177    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"]
Nov 03 00:37:19.897469 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:37:19.897432    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:37:20.282811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:20.282709    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql" event={"ID":"81963819-4fe1-4e23-8c88-908fa1e21a62","Type":"ContainerStarted","Data":"ee593e69ef3cdff3de0f489ac65200f459659edfb644b5f47f7f2c047d1a5791"}
Nov 03 00:37:20.282811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:20.282768    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql" event={"ID":"81963819-4fe1-4e23-8c88-908fa1e21a62","Type":"ContainerStarted","Data":"eb5c82d6b3dba172c5bf0950a1fd3e27b5ffc1c21a79000075aac8fc899b707d"}
Nov 03 00:37:20.323624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:20.323570    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql" podStartSLOduration=2.3235563040000002 podStartE2EDuration="2.323556304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:20.321093045 +0000 UTC m=+1877.876445070" watchObservedRunningTime="2026-11-03 00:37:20.323556304 +0000 UTC m=+1877.878908327"
Nov 03 00:37:20.977744 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:20.977689    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:28.977327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:28.977291    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:28.990029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:28.990006    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:29.412876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:29.412843    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:37:32.093783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:32.093748    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:37:48.521906 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.521828    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"]
Nov 03 00:37:48.528452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.528425    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.547965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.547938    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"]
Nov 03 00:37:48.552591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.552550    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"default-dockercfg-8kwzh\""
Nov 03 00:37:48.552803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.552780    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"auth-test-model-9908e8-kserve-self-signed-certs\""
Nov 03 00:37:48.586914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.586877    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-model-cache\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.587066 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.587031    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tls-certs\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.587110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.587067    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkq6r\" (UniqueName: \"kubernetes.io/projected/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-kube-api-access-fkq6r\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.587170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.587155    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tmp-dir\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.587213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.587182    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-home\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.587273 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.587212    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-dshm\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.619216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.619185    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"openshift-service-ca.crt\""
Nov 03 00:37:48.619398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.619303    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"kube-root-ca.crt\""
Nov 03 00:37:48.688704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.688672    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-model-cache\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.688881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.688766    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tls-certs\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.688881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.688797    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkq6r\" (UniqueName: \"kubernetes.io/projected/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-kube-api-access-fkq6r\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.688881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.688873    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tmp-dir\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.689098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.689065    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-home\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.689186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.689098    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-model-cache\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.689186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.689153    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-dshm\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.689401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.689382    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tmp-dir\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.689464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.689421    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-home\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.691512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.691489    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tls-certs\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.691627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.691526    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-dshm\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.707128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.707106    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkq6r\" (UniqueName: \"kubernetes.io/projected/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-kube-api-access-fkq6r\") pod \"auth-test-model-9908e8-kserve-7f544b64b9-sfwm7\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:48.857991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:48.856616    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:49.248435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:49.248403    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"]
Nov 03 00:37:49.610175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:49.610142    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7" event={"ID":"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b","Type":"ContainerStarted","Data":"7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b"}
Nov 03 00:37:49.610626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:49.610191    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7" event={"ID":"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b","Type":"ContainerStarted","Data":"37038224bbec1ebdbdeaf7dd5113c8d2c197668edf7cd9592d08443a07f9242a"}
Nov 03 00:37:49.649778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:49.649728    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7" podStartSLOduration=1.64971417 podStartE2EDuration="1.64971417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:37:49.64628086 +0000 UTC m=+1907.201632886" watchObservedRunningTime="2026-11-03 00:37:49.64971417 +0000 UTC m=+1907.205066193"
Nov 03 00:37:49.857674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:49.857645    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:49.898791 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:37:49.898705    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:37:57.110543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:57.110501    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"]
Nov 03 00:37:57.111218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:57.111187    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" containerID="cri-o://8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b" gracePeriod=30
Nov 03 00:37:58.381281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:58.379555    6000 patch_prober.go:29] interesting pod/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:37:58.381281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:58.379627    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:37:58.857412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:58.857374    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:58.870577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:58.870552    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:37:59.722550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:37:59.722514    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:38:02.410357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.410331    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:38:02.502770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.502741    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-envoy\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.502959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.502811    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-socket\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.502959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.502859    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-credential-socket\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.502959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.502942    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istiod-ca-cert\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.503135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.503008    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-kube-api-access-5p8vm\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-kube-api-access-5p8vm\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.503135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.503075    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-token\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-token\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.503135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.503108    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-podinfo\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.503337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.503156    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-socket" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.503448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.503427    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-credential-socket" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.503517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.503495    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-data\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.503585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.503558    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-certs\") pod \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\" (UID: \"412aff7a-a67a-4e5c-8e4e-fcb1dadea987\") "
Nov 03 00:38:02.504442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.504212    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-data" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.504442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.504351    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istiod-ca-cert" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:38:02.504628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.504478    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-certs" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.504878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.504835    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.504878 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.504872    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.505020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.504886    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.506164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.506138    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-podinfo" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:38:02.506283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.506210    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-token" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:38:02.507678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.506725    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-envoy" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:38:02.507678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.506846    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-kube-api-access-5p8vm" pod "412aff7a-a67a-4e5c-8e4e-fcb1dadea987" (UID: "412aff7a-a67a-4e5c-8e4e-fcb1dadea987"). InnerVolumeSpecName "kube-api-access-5p8vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:38:02.607230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.607202    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.607230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.607228    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.607230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.607237    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.607486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.607246    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p8vm\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-kube-api-access-5p8vm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.607486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.607272    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.607486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.607281    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/412aff7a-a67a-4e5c-8e4e-fcb1dadea987-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:38:02.733061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.732978    6000 generic.go:356] "Generic (PLEG): container finished" podID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerID="8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b" exitCode=0
Nov 03 00:38:02.733061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.733044    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"
Nov 03 00:38:02.733250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.733056    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" event={"ID":"412aff7a-a67a-4e5c-8e4e-fcb1dadea987","Type":"ContainerDied","Data":"8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b"}
Nov 03 00:38:02.733250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.733133    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl" event={"ID":"412aff7a-a67a-4e5c-8e4e-fcb1dadea987","Type":"ContainerDied","Data":"7cdf73962872ad8c0a90a8ccdc8d76151ac4527971e9741d017e084fdec4d209"}
Nov 03 00:38:02.733250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.733155    6000 scope.go:122] "RemoveContainer" containerID="8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b"
Nov 03 00:38:02.743919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.743899    6000 scope.go:122] "RemoveContainer" containerID="8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b"
Nov 03 00:38:02.744210 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:38:02.744190    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b\": container with ID starting with 8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b not found: ID does not exist" containerID="8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b"
Nov 03 00:38:02.744278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.744219    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b"} err="failed to get container status \"8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b\": rpc error: code = NotFound desc = could not find container \"8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b\": container with ID starting with 8951356988e26a7f64a44850ef8b6c20eecb6b947f315880721f3fbf944d851b not found: ID does not exist"
Nov 03 00:38:02.836281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.832320    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"]
Nov 03 00:38:02.849280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:02.848891    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-c59124-openshift-default-9b58f8dc-sbqdl"]
Nov 03 00:38:04.568245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:04.568215    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" path="/var/lib/kubelet/pods/412aff7a-a67a-4e5c-8e4e-fcb1dadea987/volumes"
Nov 03 00:38:08.543277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.543228    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"]
Nov 03 00:38:08.544296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.544244    6000 cpu_manager.go:414] "containerMap: removing container" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy"
Nov 03 00:38:08.544296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.544285    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy"
Nov 03 00:38:08.544497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.544480    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="412aff7a-a67a-4e5c-8e4e-fcb1dadea987" containerName="istio-proxy"
Nov 03 00:38:08.549666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.549646    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.572807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.572780    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"]
Nov 03 00:38:08.576397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.576370    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a07a90\"/\"kube-root-ca.crt\""
Nov 03 00:38:08.576746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.576726    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-a07a90\"/\"openshift-service-ca.crt\""
Nov 03 00:38:08.577886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.577862    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a07a90\"/\"auth-test-model-a07a90-kserve-self-signed-certs\""
Nov 03 00:38:08.578003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.577889    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-a07a90\"/\"default-dockercfg-d8khl\""
Nov 03 00:38:08.600854 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.600826    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-model-cache\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.601106 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.601059    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tls-certs\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.601287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.601132    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-kube-api-access-p469w\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.601287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.601198    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-home\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.601287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.601283    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tmp-dir\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.601565 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.601474    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-dshm\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.702748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.702714    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-dshm\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.702974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.702787    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-model-cache\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.702974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.702816    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tls-certs\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.702974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.702886    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-kube-api-access-p469w\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.702974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.702959    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-home\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.703223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.703033    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tmp-dir\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.703306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.703249    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-model-cache\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.703529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.703504    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tmp-dir\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.703529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.703521    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-home\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.705282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.705241    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-dshm\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.705592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.705575    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tls-certs\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.721726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.721700    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-kube-api-access-p469w\") pod \"auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") " pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:08.866644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:08.866606    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:09.245278 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:38:09.245232    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46b6faa9_61dc_4cbc_ab8d_46d19bc3cedc.slice/crio-dcbd924dd2edde87a05b822b4f0d49660951246c44cf2615d5d75e294ef60be1 WatchSource:0}: Error finding container dcbd924dd2edde87a05b822b4f0d49660951246c44cf2615d5d75e294ef60be1: Status 404 returned error can't find the container with id dcbd924dd2edde87a05b822b4f0d49660951246c44cf2615d5d75e294ef60be1
Nov 03 00:38:09.246179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:09.246155    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"]
Nov 03 00:38:10.072739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:10.072701    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd" event={"ID":"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc","Type":"ContainerStarted","Data":"70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3"}
Nov 03 00:38:10.072739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:10.072748    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd" event={"ID":"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc","Type":"ContainerStarted","Data":"dcbd924dd2edde87a05b822b4f0d49660951246c44cf2615d5d75e294ef60be1"}
Nov 03 00:38:10.111743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:10.111686    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd" podStartSLOduration=2.111672268 podStartE2EDuration="2.111672268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:10.11078027 +0000 UTC m=+1927.666132318" watchObservedRunningTime="2026-11-03 00:38:10.111672268 +0000 UTC m=+1927.667024291"
Nov 03 00:38:10.870523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:10.870490    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:18.871040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:18.871004    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:18.885867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:18.885831    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:19.177088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:19.177000    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:38:19.897125 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:38:19.897093    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:38:31.968773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:31.968739    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:38:49.897288 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:38:49.897203    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:38:58.684891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.684854    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"]
Nov 03 00:38:58.690591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.690570    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.718200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.718145    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b43332\"/\"kube-root-ca.crt\""
Nov 03 00:38:58.721063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.721037    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"]
Nov 03 00:38:58.726736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.726718    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-b43332\"/\"openshift-service-ca.crt\""
Nov 03 00:38:58.731551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.731530    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b43332\"/\"default-dockercfg-qqwr2\""
Nov 03 00:38:58.731653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.731601    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-b43332\"/\"explicit-model-b43332-kserve-self-signed-certs\""
Nov 03 00:38:58.798538 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.798484    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05622e23-c144-4770-b09a-95678332c26f-tls-certs\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.798923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.798886    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-tmp-dir\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.799172 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.799154    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-home\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.799367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.799349    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-model-cache\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.799598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.799550    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-dshm\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.800105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.800081    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx98q\" (UniqueName: \"kubernetes.io/projected/05622e23-c144-4770-b09a-95678332c26f-kube-api-access-tx98q\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910311    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05622e23-c144-4770-b09a-95678332c26f-tls-certs\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910359    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-tmp-dir\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910444    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-home\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910492    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-model-cache\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910562    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-dshm\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910654    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx98q\" (UniqueName: \"kubernetes.io/projected/05622e23-c144-4770-b09a-95678332c26f-kube-api-access-tx98q\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910792    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-home\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910846    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-tmp-dir\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.910900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.910890    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-model-cache\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.913127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.913099    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-dshm\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.913329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.913307    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05622e23-c144-4770-b09a-95678332c26f-tls-certs\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:58.929420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:58.929394    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx98q\" (UniqueName: \"kubernetes.io/projected/05622e23-c144-4770-b09a-95678332c26f-kube-api-access-tx98q\") pod \"explicit-model-b43332-kserve-7c7c5fc957-nw8hk\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") " pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:59.015330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:59.014500    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:38:59.435948 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:38:59.435917    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05622e23_c144_4770_b09a_95678332c26f.slice/crio-d7872c4fa6fe5de3f2069052650dc623d7fd0e7e200ac2d9ce51baecfd48575f WatchSource:0}: Error finding container d7872c4fa6fe5de3f2069052650dc623d7fd0e7e200ac2d9ce51baecfd48575f: Status 404 returned error can't find the container with id d7872c4fa6fe5de3f2069052650dc623d7fd0e7e200ac2d9ce51baecfd48575f
Nov 03 00:38:59.436107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:59.436087    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"]
Nov 03 00:38:59.736485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:59.736396    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk" event={"ID":"05622e23-c144-4770-b09a-95678332c26f","Type":"ContainerStarted","Data":"b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588"}
Nov 03 00:38:59.736485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:59.736443    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk" event={"ID":"05622e23-c144-4770-b09a-95678332c26f","Type":"ContainerStarted","Data":"d7872c4fa6fe5de3f2069052650dc623d7fd0e7e200ac2d9ce51baecfd48575f"}
Nov 03 00:38:59.777183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:38:59.777103    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk" podStartSLOduration=1.777085322 podStartE2EDuration="1.777085322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:38:59.773106278 +0000 UTC m=+1977.328458302" watchObservedRunningTime="2026-11-03 00:38:59.777085322 +0000 UTC m=+1977.332437346"
Nov 03 00:39:00.018275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:00.018106    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:39:09.016133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:09.016099    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:39:09.029049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:09.029022    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:39:09.821900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:09.821870    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:39:12.949511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:12.949292    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"]
Nov 03 00:39:19.907289 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:39:19.907235    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:39:20.569707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:20.569677    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:39:27.975958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:27.975904    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk" podUID="05622e23-c144-4770-b09a-95678332c26f" containerName="main" containerID="cri-o://b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588" gracePeriod=45
Nov 03 00:39:28.253451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.253423    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:39:28.344605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.344570    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05622e23-c144-4770-b09a-95678332c26f-kube-api-access-tx98q\" (UniqueName: \"kubernetes.io/projected/05622e23-c144-4770-b09a-95678332c26f-kube-api-access-tx98q\") pod \"05622e23-c144-4770-b09a-95678332c26f\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") "
Nov 03 00:39:28.344807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.344698    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-tmp-dir\") pod \"05622e23-c144-4770-b09a-95678332c26f\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") "
Nov 03 00:39:28.344884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.344831    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-model-cache\") pod \"05622e23-c144-4770-b09a-95678332c26f\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") "
Nov 03 00:39:28.344949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.344893    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-home\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-home\") pod \"05622e23-c144-4770-b09a-95678332c26f\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") "
Nov 03 00:39:28.345010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.344965    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05622e23-c144-4770-b09a-95678332c26f-tls-certs\" (UniqueName: \"kubernetes.io/secret/05622e23-c144-4770-b09a-95678332c26f-tls-certs\") pod \"05622e23-c144-4770-b09a-95678332c26f\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") "
Nov 03 00:39:28.345071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.345003    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-tmp-dir" pod "05622e23-c144-4770-b09a-95678332c26f" (UID: "05622e23-c144-4770-b09a-95678332c26f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:28.345071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.345062    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-dshm\") pod \"05622e23-c144-4770-b09a-95678332c26f\" (UID: \"05622e23-c144-4770-b09a-95678332c26f\") "
Nov 03 00:39:28.345180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.345148    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-model-cache" pod "05622e23-c144-4770-b09a-95678332c26f" (UID: "05622e23-c144-4770-b09a-95678332c26f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:28.345244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.345224    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-home" pod "05622e23-c144-4770-b09a-95678332c26f" (UID: "05622e23-c144-4770-b09a-95678332c26f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:28.345734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.345712    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:39:28.345734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.345733    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:39:28.345923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.345742    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:39:28.348103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.348081    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-dshm" pod "05622e23-c144-4770-b09a-95678332c26f" (UID: "05622e23-c144-4770-b09a-95678332c26f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:39:28.348193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.348168    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05622e23-c144-4770-b09a-95678332c26f-kube-api-access-tx98q" pod "05622e23-c144-4770-b09a-95678332c26f" (UID: "05622e23-c144-4770-b09a-95678332c26f"). InnerVolumeSpecName "kube-api-access-tx98q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:39:28.350390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.350367    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05622e23-c144-4770-b09a-95678332c26f-tls-certs" pod "05622e23-c144-4770-b09a-95678332c26f" (UID: "05622e23-c144-4770-b09a-95678332c26f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:39:28.447121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.447087    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05622e23-c144-4770-b09a-95678332c26f-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:39:28.447121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.447113    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05622e23-c144-4770-b09a-95678332c26f-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:39:28.447121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.447124    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx98q\" (UniqueName: \"kubernetes.io/projected/05622e23-c144-4770-b09a-95678332c26f-kube-api-access-tx98q\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:39:28.956673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.956636    6000 generic.go:356] "Generic (PLEG): container finished" podID="05622e23-c144-4770-b09a-95678332c26f" containerID="b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588" exitCode=0
Nov 03 00:39:28.956868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.956704    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"
Nov 03 00:39:28.956868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.956717    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk" event={"ID":"05622e23-c144-4770-b09a-95678332c26f","Type":"ContainerDied","Data":"b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588"}
Nov 03 00:39:28.956868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.956767    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk" event={"ID":"05622e23-c144-4770-b09a-95678332c26f","Type":"ContainerDied","Data":"d7872c4fa6fe5de3f2069052650dc623d7fd0e7e200ac2d9ce51baecfd48575f"}
Nov 03 00:39:28.956868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.956786    6000 scope.go:122] "RemoveContainer" containerID="b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588"
Nov 03 00:39:28.967363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.967345    6000 scope.go:122] "RemoveContainer" containerID="b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588"
Nov 03 00:39:28.967655 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:39:28.967637    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588\": container with ID starting with b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588 not found: ID does not exist" containerID="b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588"
Nov 03 00:39:28.967722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:28.967668    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588"} err="failed to get container status \"b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588\": rpc error: code = NotFound desc = could not find container \"b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588\": container with ID starting with b03e82c82636bea19ec81fcb16cfbb711e2c5f423960766b2eda1996045e1588 not found: ID does not exist"
Nov 03 00:39:29.006562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:29.006530    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"]
Nov 03 00:39:29.018028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:29.018000    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-b43332/explicit-model-b43332-kserve-7c7c5fc957-nw8hk"]
Nov 03 00:39:30.569858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:30.569828    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05622e23-c144-4770-b09a-95678332c26f" path="/var/lib/kubelet/pods/05622e23-c144-4770-b09a-95678332c26f/volumes"
Nov 03 00:39:49.897528 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:39:49.897493    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:39:58.071082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:39:58.071043    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:40:17.887131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:17.887093    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-78cfcc"]
Nov 03 00:40:17.887867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:17.887851    6000 cpu_manager.go:414] "containerMap: removing container" podUID="05622e23-c144-4770-b09a-95678332c26f" containerName="main"
Nov 03 00:40:17.887917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:17.887868    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05622e23-c144-4770-b09a-95678332c26f" containerName="main"
Nov 03 00:40:17.888033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:17.888023    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="05622e23-c144-4770-b09a-95678332c26f" containerName="main"
Nov 03 00:40:17.892968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:17.892950    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:17.906419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:17.906394    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-78cfcc"]
Nov 03 00:40:17.914470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:17.914445    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5fmb\""
Nov 03 00:40:18.020198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:18.020163    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrcg7\" (UniqueName: \"kubernetes.io/projected/fbf64306-d7c9-45dc-a4b5-9f71642c7612-kube-api-access-xrcg7\") pod \"test-curl-7609-78cfcc\" (UID: \"fbf64306-d7c9-45dc-a4b5-9f71642c7612\") " pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:18.121949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:18.121913    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrcg7\" (UniqueName: \"kubernetes.io/projected/fbf64306-d7c9-45dc-a4b5-9f71642c7612-kube-api-access-xrcg7\") pod \"test-curl-7609-78cfcc\" (UID: \"fbf64306-d7c9-45dc-a4b5-9f71642c7612\") " pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:18.140499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:18.140431    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrcg7\" (UniqueName: \"kubernetes.io/projected/fbf64306-d7c9-45dc-a4b5-9f71642c7612-kube-api-access-xrcg7\") pod \"test-curl-7609-78cfcc\" (UID: \"fbf64306-d7c9-45dc-a4b5-9f71642c7612\") " pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:18.209326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:18.209295    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:18.519496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:18.519466    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-78cfcc"]
Nov 03 00:40:18.519707 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:40:18.519684    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfbf64306_d7c9_45dc_a4b5_9f71642c7612.slice/crio-bb48da18bfaff21ab7b2f505ff33224f76bf0a97dcb469df8de9b1a73f393844 WatchSource:0}: Error finding container bb48da18bfaff21ab7b2f505ff33224f76bf0a97dcb469df8de9b1a73f393844: Status 404 returned error can't find the container with id bb48da18bfaff21ab7b2f505ff33224f76bf0a97dcb469df8de9b1a73f393844
Nov 03 00:40:19.423660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:19.423619    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-78cfcc" event={"ID":"fbf64306-d7c9-45dc-a4b5-9f71642c7612","Type":"ContainerStarted","Data":"92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360"}
Nov 03 00:40:19.424108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:19.423675    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-78cfcc" event={"ID":"fbf64306-d7c9-45dc-a4b5-9f71642c7612","Type":"ContainerStarted","Data":"bb48da18bfaff21ab7b2f505ff33224f76bf0a97dcb469df8de9b1a73f393844"}
Nov 03 00:40:19.459813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:19.459764    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-78cfcc" podStartSLOduration=2.459751573 podStartE2EDuration="2.459751573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:19.45847797 +0000 UTC m=+2057.013829994" watchObservedRunningTime="2026-11-03 00:40:19.459751573 +0000 UTC m=+2057.015103597"
Nov 03 00:40:19.897930 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:40:19.897890    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:40:20.359309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:20.351669    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-78cfcc"]
Nov 03 00:40:20.359309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:20.357848    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-78cfcc"]
Nov 03 00:40:21.520115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:21.520069    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-78cfcc" podUID="fbf64306-d7c9-45dc-a4b5-9f71642c7612" containerName="test-curl-7609-78cfcc" containerID="cri-o://92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360" gracePeriod=2
Nov 03 00:40:23.718722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:23.718696    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:23.850443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:23.850356    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbf64306-d7c9-45dc-a4b5-9f71642c7612-kube-api-access-xrcg7\" (UniqueName: \"kubernetes.io/projected/fbf64306-d7c9-45dc-a4b5-9f71642c7612-kube-api-access-xrcg7\") pod \"fbf64306-d7c9-45dc-a4b5-9f71642c7612\" (UID: \"fbf64306-d7c9-45dc-a4b5-9f71642c7612\") "
Nov 03 00:40:23.854730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:23.854698    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbf64306-d7c9-45dc-a4b5-9f71642c7612-kube-api-access-xrcg7" pod "fbf64306-d7c9-45dc-a4b5-9f71642c7612" (UID: "fbf64306-d7c9-45dc-a4b5-9f71642c7612"). InnerVolumeSpecName "kube-api-access-xrcg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:23.953248 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:23.953215    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrcg7\" (UniqueName: \"kubernetes.io/projected/fbf64306-d7c9-45dc-a4b5-9f71642c7612-kube-api-access-xrcg7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:24.548109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:24.548023    6000 generic.go:356] "Generic (PLEG): container finished" podID="fbf64306-d7c9-45dc-a4b5-9f71642c7612" containerID="92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360" exitCode=137
Nov 03 00:40:24.548109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:24.548073    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:24.548401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:24.548122    6000 scope.go:122] "RemoveContainer" containerID="92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360"
Nov 03 00:40:24.574084 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:40:24.574041    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7609-78cfcc\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="fbf64306-d7c9-45dc-a4b5-9f71642c7612" pod="opendatahub/test-curl-7609-78cfcc"
Nov 03 00:40:24.578197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:24.578164    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbf64306-d7c9-45dc-a4b5-9f71642c7612" path="/var/lib/kubelet/pods/fbf64306-d7c9-45dc-a4b5-9f71642c7612/volumes"
Nov 03 00:40:24.583479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:24.583455    6000 scope.go:122] "RemoveContainer" containerID="92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360"
Nov 03 00:40:24.583863 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:40:24.583837    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360\": container with ID starting with 92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360 not found: ID does not exist" containerID="92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360"
Nov 03 00:40:24.583977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:24.583872    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360"} err="failed to get container status \"92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360\": rpc error: code = NotFound desc = could not find container \"92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360\": container with ID starting with 92941eb2938bc280840dae8d672b2ee1a4f2913d68bfe621ecfbd6010a603360 not found: ID does not exist"
Nov 03 00:40:27.115203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.115171    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"]
Nov 03 00:40:27.115966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.115950    6000 cpu_manager.go:414] "containerMap: removing container" podUID="fbf64306-d7c9-45dc-a4b5-9f71642c7612" containerName="test-curl-7609-78cfcc"
Nov 03 00:40:27.115966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.115966    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbf64306-d7c9-45dc-a4b5-9f71642c7612" containerName="test-curl-7609-78cfcc"
Nov 03 00:40:27.116097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.116078    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbf64306-d7c9-45dc-a4b5-9f71642c7612" containerName="test-curl-7609-78cfcc"
Nov 03 00:40:27.120707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.120688    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.141985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.141957    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"]
Nov 03 00:40:27.147553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.147526    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-099583-dockercfg-khxjt\""
Nov 03 00:40:27.168102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.166215    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc4g9\" (UniqueName: \"kubernetes.io/projected/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-kube-api-access-cc4g9\") pod \"payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.168102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.166524    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.172475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.170093    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-099583\""
Nov 03 00:40:27.289275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.280516    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.289275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.280778    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc4g9\" (UniqueName: \"kubernetes.io/projected/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-kube-api-access-cc4g9\") pod \"payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.289275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.282084    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.364289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.358593    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc4g9\" (UniqueName: \"kubernetes.io/projected/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-kube-api-access-cc4g9\") pod \"payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.482831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.482730    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:27.818354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:27.818158    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"]
Nov 03 00:40:28.623009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:28.622973    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" event={"ID":"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f","Type":"ContainerStarted","Data":"a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66"}
Nov 03 00:40:28.623541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:28.623514    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" event={"ID":"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f","Type":"ContainerStarted","Data":"a62680d345b8853bdde7a828cc351341d453c85da35970d12705c9f41c6d2b91"}
Nov 03 00:40:28.624430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:28.624410    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:28.672845 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:28.672790    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" podStartSLOduration=1.672775401 podStartE2EDuration="1.672775401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:40:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:40:28.666619088 +0000 UTC m=+2066.221971113" watchObservedRunningTime="2026-11-03 00:40:28.672775401 +0000 UTC m=+2066.228127451"
Nov 03 00:40:30.650703 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:30.650669    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"]
Nov 03 00:40:36.877055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:36.877017    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"]
Nov 03 00:40:36.877660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:36.877286    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy" containerID="cri-o://44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b" gracePeriod=30
Nov 03 00:40:40.673115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:40.673087    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 00:40:42.457702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.457677    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:40:42.563872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.563828    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-credential-socket\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.563961    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-podinfo\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.564040    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-kube-api-access-fr2qj\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-kube-api-access-fr2qj\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.564192    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-socket\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.564277    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-data\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.564330    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istiod-ca-cert\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.564514    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-certs\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.564621    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-token\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-token\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.564869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.564840    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-envoy\") pod \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\" (UID: \"b4f1e4dd-5085-4b91-91af-cdffe80186c0\") "
Nov 03 00:40:42.566713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.566682    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-socket" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.570083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.569027    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-certs" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.570083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.569376    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.570083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.569450    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-data" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.571227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.570514    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istiod-ca-cert" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:40:42.576618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.576500    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-kube-api-access-fr2qj" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "kube-api-access-fr2qj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.576987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.576959    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-credential-socket" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.577479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.577452    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-token" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:42.587327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.587299    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-podinfo" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:40:42.587457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.587341    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-envoy" pod "b4f1e4dd-5085-4b91-91af-cdffe80186c0" (UID: "b4f1e4dd-5085-4b91-91af-cdffe80186c0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:42.672050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672013    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.672050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672052    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.672314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672096    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr2qj\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-kube-api-access-fr2qj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.672314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672114    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.672314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672146    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.672314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672160    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.672314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672172    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.672314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.672184    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4f1e4dd-5085-4b91-91af-cdffe80186c0-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:42.779097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.779055    6000 generic.go:356] "Generic (PLEG): container finished" podID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerID="44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b" exitCode=0
Nov 03 00:40:42.779312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.779135    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"
Nov 03 00:40:42.779312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.779141    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" event={"ID":"b4f1e4dd-5085-4b91-91af-cdffe80186c0","Type":"ContainerDied","Data":"44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b"}
Nov 03 00:40:42.779312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.779216    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph" event={"ID":"b4f1e4dd-5085-4b91-91af-cdffe80186c0","Type":"ContainerDied","Data":"f0525efa29ab9fbce9b0024447892c3551926893bc26673aa2d2777d463e5281"}
Nov 03 00:40:42.779312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.779267    6000 scope.go:122] "RemoveContainer" containerID="44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b"
Nov 03 00:40:42.790650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.790633    6000 scope.go:122] "RemoveContainer" containerID="44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b"
Nov 03 00:40:42.791246 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:40:42.791220    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b\": container with ID starting with 44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b not found: ID does not exist" containerID="44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b"
Nov 03 00:40:42.791432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.791407    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b"} err="failed to get container status \"44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b\": rpc error: code = NotFound desc = could not find container \"44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b\": container with ID starting with 44a054a451f457f4f88b922d0deb8f921acaf97bbe2eb3c7164d4573ca1b1e9b not found: ID does not exist"
Nov 03 00:40:42.830216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.830139    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"]
Nov 03 00:40:42.841128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:42.841098    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-4pcph"]
Nov 03 00:40:44.569342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:44.569302    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" path="/var/lib/kubelet/pods/b4f1e4dd-5085-4b91-91af-cdffe80186c0/volumes"
Nov 03 00:40:45.683289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:45.683194    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql" podUID="81963819-4fe1-4e23-8c88-908fa1e21a62" containerName="main" containerID="cri-o://ee593e69ef3cdff3de0f489ac65200f459659edfb644b5f47f7f2c047d1a5791" gracePeriod=45
Nov 03 00:40:45.805335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:45.805304    6000 generic.go:356] "Generic (PLEG): container finished" podID="81963819-4fe1-4e23-8c88-908fa1e21a62" containerID="ee593e69ef3cdff3de0f489ac65200f459659edfb644b5f47f7f2c047d1a5791" exitCode=0
Nov 03 00:40:45.805449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:45.805371    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql" event={"ID":"81963819-4fe1-4e23-8c88-908fa1e21a62","Type":"ContainerDied","Data":"ee593e69ef3cdff3de0f489ac65200f459659edfb644b5f47f7f2c047d1a5791"}
Nov 03 00:40:45.991816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:45.991790    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:40:46.137691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.137664    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81963819-4fe1-4e23-8c88-908fa1e21a62-tls-certs\" (UniqueName: \"kubernetes.io/secret/81963819-4fe1-4e23-8c88-908fa1e21a62-tls-certs\") pod \"81963819-4fe1-4e23-8c88-908fa1e21a62\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") "
Nov 03 00:40:46.137859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.137776    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81963819-4fe1-4e23-8c88-908fa1e21a62-kube-api-access-p78nt\" (UniqueName: \"kubernetes.io/projected/81963819-4fe1-4e23-8c88-908fa1e21a62-kube-api-access-p78nt\") pod \"81963819-4fe1-4e23-8c88-908fa1e21a62\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") "
Nov 03 00:40:46.137859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.137834    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-model-cache\") pod \"81963819-4fe1-4e23-8c88-908fa1e21a62\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") "
Nov 03 00:40:46.137942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.137870    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-dshm\") pod \"81963819-4fe1-4e23-8c88-908fa1e21a62\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") "
Nov 03 00:40:46.138010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.137993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-home\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-home\") pod \"81963819-4fe1-4e23-8c88-908fa1e21a62\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") "
Nov 03 00:40:46.138142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.138118    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-model-cache" pod "81963819-4fe1-4e23-8c88-908fa1e21a62" (UID: "81963819-4fe1-4e23-8c88-908fa1e21a62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:46.138311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.138291    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-tmp-dir\") pod \"81963819-4fe1-4e23-8c88-908fa1e21a62\" (UID: \"81963819-4fe1-4e23-8c88-908fa1e21a62\") "
Nov 03 00:40:46.138402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.138302    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-home" pod "81963819-4fe1-4e23-8c88-908fa1e21a62" (UID: "81963819-4fe1-4e23-8c88-908fa1e21a62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:46.138792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.138759    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-tmp-dir" pod "81963819-4fe1-4e23-8c88-908fa1e21a62" (UID: "81963819-4fe1-4e23-8c88-908fa1e21a62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:46.139344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.139324    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:46.139388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.139351    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:46.139388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.139365    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:46.140274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.140235    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81963819-4fe1-4e23-8c88-908fa1e21a62-tls-certs" pod "81963819-4fe1-4e23-8c88-908fa1e21a62" (UID: "81963819-4fe1-4e23-8c88-908fa1e21a62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:40:46.140451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.140429    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-dshm" pod "81963819-4fe1-4e23-8c88-908fa1e21a62" (UID: "81963819-4fe1-4e23-8c88-908fa1e21a62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:40:46.140451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.140440    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81963819-4fe1-4e23-8c88-908fa1e21a62-kube-api-access-p78nt" pod "81963819-4fe1-4e23-8c88-908fa1e21a62" (UID: "81963819-4fe1-4e23-8c88-908fa1e21a62"). InnerVolumeSpecName "kube-api-access-p78nt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:40:46.241399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.241313    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p78nt\" (UniqueName: \"kubernetes.io/projected/81963819-4fe1-4e23-8c88-908fa1e21a62-kube-api-access-p78nt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:46.241399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.241340    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81963819-4fe1-4e23-8c88-908fa1e21a62-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:46.241399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.241351    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81963819-4fe1-4e23-8c88-908fa1e21a62-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:40:46.813146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.813113    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"
Nov 03 00:40:46.813146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.813135    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql" event={"ID":"81963819-4fe1-4e23-8c88-908fa1e21a62","Type":"ContainerDied","Data":"eb5c82d6b3dba172c5bf0950a1fd3e27b5ffc1c21a79000075aac8fc899b707d"}
Nov 03 00:40:46.813685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.813185    6000 scope.go:122] "RemoveContainer" containerID="ee593e69ef3cdff3de0f489ac65200f459659edfb644b5f47f7f2c047d1a5791"
Nov 03 00:40:46.862799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.862775    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"]
Nov 03 00:40:46.876196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:46.876163    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-7ef1b6/auth-test-model-7ef1b6-kserve-5f4dfdc49-km4ql"]
Nov 03 00:40:48.569040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:48.569008    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81963819-4fe1-4e23-8c88-908fa1e21a62" path="/var/lib/kubelet/pods/81963819-4fe1-4e23-8c88-908fa1e21a62/volumes"
Nov 03 00:40:49.898391 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:40:49.898356    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:40:57.383960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:40:57.383918    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:41:06.878751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:06.878712    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"]
Nov 03 00:41:06.879246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:06.878995    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy" containerID="cri-o://b24bfdd53823c1c4e3c2460224549b70ec0aae11ee41d8108189ddfc90067729" gracePeriod=30
Nov 03 00:41:06.977140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:06.977101    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"]
Nov 03 00:41:06.977363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:06.977303    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy" containerID="cri-o://9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6" gracePeriod=30
Nov 03 00:41:07.037147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.031428    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"]
Nov 03 00:41:07.037147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.031676    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy" containerID="cri-o://b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2" gracePeriod=30
Nov 03 00:41:07.068713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.068657    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"]
Nov 03 00:41:07.069174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.069143    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy" containerID="cri-o://44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693" gracePeriod=30
Nov 03 00:41:07.075705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.075665    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"]
Nov 03 00:41:07.076078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.076045    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy" containerID="cri-o://154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f" gracePeriod=30
Nov 03 00:41:07.085788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.085757    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"]
Nov 03 00:41:07.086062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.086034    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy" containerID="cri-o://25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333" gracePeriod=30
Nov 03 00:41:07.094368 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.094339    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"]
Nov 03 00:41:07.094587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:07.094559    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy" containerID="cri-o://ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e" gracePeriod=30
Nov 03 00:41:10.285116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:10.285078    6000 patch_prober.go:29] interesting pod/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:41:10.285597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:10.285148    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:41:12.092674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.092644    6000 generic.go:356] "Generic (PLEG): container finished" podID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerID="b24bfdd53823c1c4e3c2460224549b70ec0aae11ee41d8108189ddfc90067729" exitCode=0
Nov 03 00:41:12.093113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.092805    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" event={"ID":"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc","Type":"ContainerDied","Data":"b24bfdd53823c1c4e3c2460224549b70ec0aae11ee41d8108189ddfc90067729"}
Nov 03 00:41:12.378113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.378085    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:41:12.381909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.381836    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:41:12.415672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.415642    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:41:12.459953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.459921    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-socket\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.460140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.459970    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-podinfo\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.460140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460020    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-podinfo\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.460140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460084    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-certs\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.460140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460124    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-envoy\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.460424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460171    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-token\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-token\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.460424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460214    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-istio-token\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-istio-token\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.460424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460249    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-certs\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.460424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460362    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-envoy\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.460424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460396    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-certs\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.460424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460423    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c572c65-36af-43d5-98e1-7cf57034a71f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c572c65-36af-43d5-98e1-7cf57034a71f-istiod-ca-cert\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.460758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460453    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-data\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.460758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460518    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-kube-api-access-5g9pc\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-kube-api-access-5g9pc\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.460758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460577    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-credential-socket\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.460758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460600    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-socket" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.460758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460640    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istiod-ca-cert\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.460758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460684    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-token\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-token\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.460758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460713    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-socket\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.460766    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-credential-socket\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461110    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86096906-704d-455f-aab8-4b852fb2bba6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86096906-704d-455f-aab8-4b852fb2bba6-istiod-ca-cert\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461170    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-data\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461201    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-data\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461559    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-credential-socket\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461620    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-envoy\") pod \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\" (UID: \"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461650    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-kube-api-access-vxhq4\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-kube-api-access-vxhq4\") pod \"7c572c65-36af-43d5-98e1-7cf57034a71f\" (UID: \"7c572c65-36af-43d5-98e1-7cf57034a71f\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461748    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-socket\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461794    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/86096906-704d-455f-aab8-4b852fb2bba6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86096906-704d-455f-aab8-4b852fb2bba6-istio-podinfo\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.461941    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-kube-api-access-wsq9m\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-kube-api-access-wsq9m\") pod \"86096906-704d-455f-aab8-4b852fb2bba6\" (UID: \"86096906-704d-455f-aab8-4b852fb2bba6\") "
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.462161    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-certs" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.462199    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-credential-socket" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.462483    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-credential-socket" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.462619    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istiod-ca-cert" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.462683    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-credential-socket" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.462811    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86096906-704d-455f-aab8-4b852fb2bba6-istiod-ca-cert" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.463021    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c572c65-36af-43d5-98e1-7cf57034a71f-istiod-ca-cert" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.463936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.463201    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-data" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.465550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.465051    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-certs" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.465550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.465107    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-certs" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.465550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.465149    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-data" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.465550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.465316    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-data" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.465550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.465513    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-socket" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.466429 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.466335    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-kube-api-access-5g9pc" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "kube-api-access-5g9pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.469543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.469473    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-socket" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.475089 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475058    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-kube-api-access-vxhq4" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "kube-api-access-vxhq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.475323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475302    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475328    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475341    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475354    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7c572c65-36af-43d5-98e1-7cf57034a71f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475368    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475384    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5g9pc\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-kube-api-access-5g9pc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475397    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475410    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475424    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475437    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475451    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/86096906-704d-455f-aab8-4b852fb2bba6-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475464    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475476    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475488    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.475745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.475502    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxhq4\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-kube-api-access-vxhq4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.476550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.476515    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-envoy" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.477031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.476996    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/86096906-704d-455f-aab8-4b852fb2bba6-istio-podinfo" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.477133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.477032    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.477133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.477057    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.477319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.477173    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-kube-api-access-wsq9m" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "kube-api-access-wsq9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.477792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.477769    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-envoy" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.479437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.479204    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-envoy" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.479721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.479432    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-token" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.479721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.479533    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-istio-token" pod "86096906-704d-455f-aab8-4b852fb2bba6" (UID: "86096906-704d-455f-aab8-4b852fb2bba6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.500944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.495059    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-podinfo" pod "7c572c65-36af-43d5-98e1-7cf57034a71f" (UID: "7c572c65-36af-43d5-98e1-7cf57034a71f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.500944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.496414    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-podinfo" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.500944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.497973    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-token" pod "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" (UID: "dc2ff918-fa0b-4719-b4f1-ed8e723c96fc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.500944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.498800    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:41:12.556411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.556384    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:41:12.566850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.566768    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:41:12.579046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.579021    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.580599    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-podinfo\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.580745    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ff2b1336-e825-47bd-b58a-a72c71d446fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2b1336-e825-47bd-b58a-a72c71d446fb-istiod-ca-cert\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.580783    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-data\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.580900    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-envoy\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.581687    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-certs\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.581735    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-socket\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.581912    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-token\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-token\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.581947    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-credential-socket\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.581993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-kube-api-access-fdb6m\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-kube-api-access-fdb6m\") pod \"ff2b1336-e825-47bd-b58a-a72c71d446fb\" (UID: \"ff2b1336-e825-47bd-b58a-a72c71d446fb\") "
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582644    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582666    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582680    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582694    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/86096906-704d-455f-aab8-4b852fb2bba6-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582706    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsq9m\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-kube-api-access-wsq9m\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582719    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582732    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582744    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/86096906-704d-455f-aab8-4b852fb2bba6-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582757    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7c572c65-36af-43d5-98e1-7cf57034a71f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.584328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.582768    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/86096906-704d-455f-aab8-4b852fb2bba6-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.592990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.592962    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-socket" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.593190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.593166    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-certs" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.593661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.593634    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff2b1336-e825-47bd-b58a-a72c71d446fb-istiod-ca-cert" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.595637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.594922    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-data" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.602551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.602517    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-kube-api-access-fdb6m" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "kube-api-access-fdb6m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.602702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.602682    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-token" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.602791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.602772    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-envoy" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.606200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.606175    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-credential-socket" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.620167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.620136    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-podinfo" pod "ff2b1336-e825-47bd-b58a-a72c71d446fb" (UID: "ff2b1336-e825-47bd-b58a-a72c71d446fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.684742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.684649    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-istio-token\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-istio-token\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.684742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.684702    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-socket\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.684988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.684872    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-envoy\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.684988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.684958    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-socket" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.684988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.684960    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-socket\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.685162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685036    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-token\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-token\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.685162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685059    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-podinfo\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.685162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685081    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-kube-api-access-hpr9h\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-kube-api-access-hpr9h\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.685162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685127    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-data\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685172    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-socket" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685180    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-credential-socket\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685239    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-envoy\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685363    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-socket\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685400    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-certs\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685435    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-token\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-token\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685487    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istiod-ca-cert\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685542    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60317f80-10cb-4053-bfec-17853dc9fada-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60317f80-10cb-4053-bfec-17853dc9fada-istiod-ca-cert\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.685605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685588    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-podinfo\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685620    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-credential-socket\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685648    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-data\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685703    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-certs\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685735    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-kube-api-access-sp8xt\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-kube-api-access-sp8xt\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685770    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istiod-ca-cert\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685831    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-envoy\") pod \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\" (UID: \"45aef524-0f0d-4db9-aece-46db1d4e6e2f\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685851    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-certs" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685862    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-credential-socket\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685912    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-data\") pod \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\" (UID: \"009cbac3-f61f-4f63-9500-8ecf3a57afb0\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685943    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-credential-socket" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685952    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-certs\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.685991    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60317f80-10cb-4053-bfec-17853dc9fada-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60317f80-10cb-4053-bfec-17853dc9fada-istio-podinfo\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.686073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.686033    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-kube-api-access-pdppb\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-kube-api-access-pdppb\") pod \"60317f80-10cb-4053-bfec-17853dc9fada\" (UID: \"60317f80-10cb-4053-bfec-17853dc9fada\") "
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.686216    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-socket" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.687898    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.687922    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.687936    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.687948    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff2b1336-e825-47bd-b58a-a72c71d446fb-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688444    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688463    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688477    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688491    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688506    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688521    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688535    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688549    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff2b1336-e825-47bd-b58a-a72c71d446fb-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688565    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688595    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdb6m\" (UniqueName: \"kubernetes.io/projected/ff2b1336-e825-47bd-b58a-a72c71d446fb-kube-api-access-fdb6m\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688086    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-data" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688187    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-certs" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688409    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-data" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688411    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60317f80-10cb-4053-bfec-17853dc9fada-istiod-ca-cert" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688464    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-credential-socket" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688831    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-credential-socket" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688877    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istiod-ca-cert" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.688988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.688884    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-certs" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.689609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.689306    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-data" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.689609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.689599    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istiod-ca-cert" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:12.693409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.693384    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-token" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.693521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.693443    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-kube-api-access-sp8xt" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "kube-api-access-sp8xt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.693580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.693543    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60317f80-10cb-4053-bfec-17853dc9fada-istio-podinfo" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.693953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.693930    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-kube-api-access-pdppb" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "kube-api-access-pdppb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.694050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.693964    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-kube-api-access-hpr9h" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "kube-api-access-hpr9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.694050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.693998    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-envoy" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.694050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.694020    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-podinfo" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.694050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.694032    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-envoy" pod "009cbac3-f61f-4f63-9500-8ecf3a57afb0" (UID: "009cbac3-f61f-4f63-9500-8ecf3a57afb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.694277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.694185    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-token" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.694669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.694648    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-podinfo" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:12.695674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.695643    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-envoy" pod "45aef524-0f0d-4db9-aece-46db1d4e6e2f" (UID: "45aef524-0f0d-4db9-aece-46db1d4e6e2f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:12.695674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.695639    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-istio-token" pod "60317f80-10cb-4053-bfec-17853dc9fada" (UID: "60317f80-10cb-4053-bfec-17853dc9fada"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:12.790200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790155    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790190    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790203    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpr9h\" (UniqueName: \"kubernetes.io/projected/45aef524-0f0d-4db9-aece-46db1d4e6e2f-kube-api-access-hpr9h\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790218    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790231    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790242    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790267    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790279    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60317f80-10cb-4053-bfec-17853dc9fada-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790290    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790301    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790311    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790323    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790335    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sp8xt\" (UniqueName: \"kubernetes.io/projected/009cbac3-f61f-4f63-9500-8ecf3a57afb0-kube-api-access-sp8xt\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790346    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790357    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45aef524-0f0d-4db9-aece-46db1d4e6e2f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790368    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790382    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/009cbac3-f61f-4f63-9500-8ecf3a57afb0-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790394    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790405    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60317f80-10cb-4053-bfec-17853dc9fada-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790418    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdppb\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-kube-api-access-pdppb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790430    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60317f80-10cb-4053-bfec-17853dc9fada-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:12.790533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:12.790443    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60317f80-10cb-4053-bfec-17853dc9fada-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:13.101209 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.101172    6000 generic.go:356] "Generic (PLEG): container finished" podID="86096906-704d-455f-aab8-4b852fb2bba6" containerID="9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6" exitCode=0
Nov 03 00:41:13.101686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.101284    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"
Nov 03 00:41:13.101686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.101295    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" event={"ID":"86096906-704d-455f-aab8-4b852fb2bba6","Type":"ContainerDied","Data":"9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6"}
Nov 03 00:41:13.101686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.101362    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp" event={"ID":"86096906-704d-455f-aab8-4b852fb2bba6","Type":"ContainerDied","Data":"b00980bacaf4b256e1fbd0a9172e6da4a70a72c2b970958b33bf1a64e64ef8bc"}
Nov 03 00:41:13.101686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.101390    6000 scope.go:122] "RemoveContainer" containerID="9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6"
Nov 03 00:41:13.103069 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.103010    6000 generic.go:356] "Generic (PLEG): container finished" podID="60317f80-10cb-4053-bfec-17853dc9fada" containerID="ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e" exitCode=0
Nov 03 00:41:13.103154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.103071    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"
Nov 03 00:41:13.103154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.103112    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" event={"ID":"60317f80-10cb-4053-bfec-17853dc9fada","Type":"ContainerDied","Data":"ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e"}
Nov 03 00:41:13.103311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.103170    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd" event={"ID":"60317f80-10cb-4053-bfec-17853dc9fada","Type":"ContainerDied","Data":"9a99e23f4e19a30b9a43f4b31d09f6b58b80b5f9fea818c1391504ff2ac9ce96"}
Nov 03 00:41:13.105091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.105062    6000 generic.go:356] "Generic (PLEG): container finished" podID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerID="154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f" exitCode=0
Nov 03 00:41:13.105250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.105119    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"
Nov 03 00:41:13.105250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.105156    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" event={"ID":"7c572c65-36af-43d5-98e1-7cf57034a71f","Type":"ContainerDied","Data":"154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f"}
Nov 03 00:41:13.105250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.105208    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b" event={"ID":"7c572c65-36af-43d5-98e1-7cf57034a71f","Type":"ContainerDied","Data":"e14a337b9532cf663e99af29624c07eaac4ef71f57de90e6373e7d7d8de31741"}
Nov 03 00:41:13.106936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.106912    6000 generic.go:356] "Generic (PLEG): container finished" podID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerID="b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2" exitCode=0
Nov 03 00:41:13.107204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.106977    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"
Nov 03 00:41:13.107204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.106995    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" event={"ID":"009cbac3-f61f-4f63-9500-8ecf3a57afb0","Type":"ContainerDied","Data":"b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2"}
Nov 03 00:41:13.107204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.107052    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh" event={"ID":"009cbac3-f61f-4f63-9500-8ecf3a57afb0","Type":"ContainerDied","Data":"66c00c46ddf28b14a68df35569ca4d47b1205329168d3f7a342f61a3875d69ce"}
Nov 03 00:41:13.110706 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.110680    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"
Nov 03 00:41:13.110706 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.110694    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd" event={"ID":"dc2ff918-fa0b-4719-b4f1-ed8e723c96fc","Type":"ContainerDied","Data":"5305eaa19769694e83eca106a5c00e3b5b17b604224a623f617d49938e90d931"}
Nov 03 00:41:13.112859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.112834    6000 generic.go:356] "Generic (PLEG): container finished" podID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerID="25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333" exitCode=0
Nov 03 00:41:13.112952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.112937    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" event={"ID":"ff2b1336-e825-47bd-b58a-a72c71d446fb","Type":"ContainerDied","Data":"25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333"}
Nov 03 00:41:13.113150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.112980    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm" event={"ID":"ff2b1336-e825-47bd-b58a-a72c71d446fb","Type":"ContainerDied","Data":"381ab81a2534d6255df63d5936e139b65ddab3dca6e4f3966dbd6efd4d490e33"}
Nov 03 00:41:13.113150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.112942    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"
Nov 03 00:41:13.114986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.114791    6000 generic.go:356] "Generic (PLEG): container finished" podID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerID="44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693" exitCode=0
Nov 03 00:41:13.114986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.114828    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" event={"ID":"45aef524-0f0d-4db9-aece-46db1d4e6e2f","Type":"ContainerDied","Data":"44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693"}
Nov 03 00:41:13.114986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.114845    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"
Nov 03 00:41:13.114986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.114864    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh" event={"ID":"45aef524-0f0d-4db9-aece-46db1d4e6e2f","Type":"ContainerDied","Data":"bf2d8f525b4ebc7c819b2c4aaf2ff30ba9d38083d2af850a01c5161bfa1fcd07"}
Nov 03 00:41:13.117279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.117220    6000 scope.go:122] "RemoveContainer" containerID="9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6"
Nov 03 00:41:13.117617 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:13.117574    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6\": container with ID starting with 9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6 not found: ID does not exist" containerID="9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6"
Nov 03 00:41:13.117733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.117624    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6"} err="failed to get container status \"9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6\": rpc error: code = NotFound desc = could not find container \"9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6\": container with ID starting with 9a21b2c7da3aa02bdc85dbc06176b6827e912a3b3a2cbfa4d9bd22bfc8bc63a6 not found: ID does not exist"
Nov 03 00:41:13.117733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.117641    6000 scope.go:122] "RemoveContainer" containerID="ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e"
Nov 03 00:41:13.130075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.130047    6000 scope.go:122] "RemoveContainer" containerID="ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e"
Nov 03 00:41:13.130371 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:13.130350    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e\": container with ID starting with ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e not found: ID does not exist" containerID="ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e"
Nov 03 00:41:13.130467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.130376    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e"} err="failed to get container status \"ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e\": rpc error: code = NotFound desc = could not find container \"ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e\": container with ID starting with ebe2c88c34587dcce47c3038d87b49111a982d1b0be036675e3d3bff48b0307e not found: ID does not exist"
Nov 03 00:41:13.130467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.130395    6000 scope.go:122] "RemoveContainer" containerID="154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f"
Nov 03 00:41:13.140461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.140444    6000 scope.go:122] "RemoveContainer" containerID="154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f"
Nov 03 00:41:13.140736 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:13.140715    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f\": container with ID starting with 154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f not found: ID does not exist" containerID="154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f"
Nov 03 00:41:13.140783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.140745    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f"} err="failed to get container status \"154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f\": rpc error: code = NotFound desc = could not find container \"154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f\": container with ID starting with 154b09412d5c92eeadc4b568e16c262c2f762de36a36b7b23418497e80ece41f not found: ID does not exist"
Nov 03 00:41:13.140783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.140761    6000 scope.go:122] "RemoveContainer" containerID="b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2"
Nov 03 00:41:13.150352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.150331    6000 scope.go:122] "RemoveContainer" containerID="b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2"
Nov 03 00:41:13.151193 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:13.151169    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2\": container with ID starting with b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2 not found: ID does not exist" containerID="b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2"
Nov 03 00:41:13.151298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.151199    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2"} err="failed to get container status \"b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2\": rpc error: code = NotFound desc = could not find container \"b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2\": container with ID starting with b831025e95d5e41a59f622d116e4c8ff2bded13d2264905c3f293511aa5d8ae2 not found: ID does not exist"
Nov 03 00:41:13.151298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.151216    6000 scope.go:122] "RemoveContainer" containerID="b24bfdd53823c1c4e3c2460224549b70ec0aae11ee41d8108189ddfc90067729"
Nov 03 00:41:13.154300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.154278    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"]
Nov 03 00:41:13.161249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.161230    6000 scope.go:122] "RemoveContainer" containerID="25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333"
Nov 03 00:41:13.165927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.165905    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-r7qxd"]
Nov 03 00:41:13.170930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.170909    6000 scope.go:122] "RemoveContainer" containerID="25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333"
Nov 03 00:41:13.171201 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:13.171181    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333\": container with ID starting with 25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333 not found: ID does not exist" containerID="25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333"
Nov 03 00:41:13.171321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.171212    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333"} err="failed to get container status \"25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333\": rpc error: code = NotFound desc = could not find container \"25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333\": container with ID starting with 25d3400ddfdd29930e501251c8eaa586a9b5cbdf26a121f9400ec092decee333 not found: ID does not exist"
Nov 03 00:41:13.171321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.171236    6000 scope.go:122] "RemoveContainer" containerID="44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693"
Nov 03 00:41:13.180974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.180955    6000 scope.go:122] "RemoveContainer" containerID="44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693"
Nov 03 00:41:13.181233 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:13.181214    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693\": container with ID starting with 44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693 not found: ID does not exist" containerID="44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693"
Nov 03 00:41:13.181324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.181244    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693"} err="failed to get container status \"44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693\": rpc error: code = NotFound desc = could not find container \"44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693\": container with ID starting with 44e5cff44bff10d7de516d7e6b90523f8a8b57259ecf9784123c011af4db3693 not found: ID does not exist"
Nov 03 00:41:13.231126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.231050    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"]
Nov 03 00:41:13.234191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.234170    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7jxwd"]
Nov 03 00:41:13.265395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.265365    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"]
Nov 03 00:41:13.268949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.268921    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-kjlpp"]
Nov 03 00:41:13.302113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.302082    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"]
Nov 03 00:41:13.305998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.305972    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-94v5b"]
Nov 03 00:41:13.343015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.342968    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"]
Nov 03 00:41:13.346691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.346665    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-ldgbh"]
Nov 03 00:41:13.379082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.379046    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"]
Nov 03 00:41:13.385609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.385580    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-6hrgm"]
Nov 03 00:41:13.417094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.417061    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"]
Nov 03 00:41:13.422061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:13.422033    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-jqlsh"]
Nov 03 00:41:14.568022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:14.567989    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" path="/var/lib/kubelet/pods/009cbac3-f61f-4f63-9500-8ecf3a57afb0/volumes"
Nov 03 00:41:14.568532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:14.568518    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" path="/var/lib/kubelet/pods/45aef524-0f0d-4db9-aece-46db1d4e6e2f/volumes"
Nov 03 00:41:14.568942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:14.568929    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60317f80-10cb-4053-bfec-17853dc9fada" path="/var/lib/kubelet/pods/60317f80-10cb-4053-bfec-17853dc9fada/volumes"
Nov 03 00:41:14.569742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:14.569687    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" path="/var/lib/kubelet/pods/7c572c65-36af-43d5-98e1-7cf57034a71f/volumes"
Nov 03 00:41:14.570111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:14.570099    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86096906-704d-455f-aab8-4b852fb2bba6" path="/var/lib/kubelet/pods/86096906-704d-455f-aab8-4b852fb2bba6/volumes"
Nov 03 00:41:14.570590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:14.570575    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" path="/var/lib/kubelet/pods/dc2ff918-fa0b-4719-b4f1-ed8e723c96fc/volumes"
Nov 03 00:41:14.571324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:14.571309    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" path="/var/lib/kubelet/pods/ff2b1336-e825-47bd-b58a-a72c71d446fb/volumes"
Nov 03 00:41:15.360868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.360839    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7615-5304f0"]
Nov 03 00:41:15.361513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361501    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy"
Nov 03 00:41:15.361567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361515    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy"
Nov 03 00:41:15.361567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361533    6000 cpu_manager.go:414] "containerMap: removing container" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy"
Nov 03 00:41:15.361567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361537    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy"
Nov 03 00:41:15.361567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361559    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy"
Nov 03 00:41:15.361567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361565    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361575    6000 cpu_manager.go:414] "containerMap: removing container" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361579    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361596    6000 cpu_manager.go:414] "containerMap: removing container" podUID="81963819-4fe1-4e23-8c88-908fa1e21a62" containerName="main"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361601    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81963819-4fe1-4e23-8c88-908fa1e21a62" containerName="main"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361610    6000 cpu_manager.go:414] "containerMap: removing container" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361614    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361630    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361634    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361639    6000 cpu_manager.go:414] "containerMap: removing container" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361644    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361656    6000 cpu_manager.go:414] "containerMap: removing container" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy"
Nov 03 00:41:15.361719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361660    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361742    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="81963819-4fe1-4e23-8c88-908fa1e21a62" containerName="main"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361753    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="009cbac3-f61f-4f63-9500-8ecf3a57afb0" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361762    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4f1e4dd-5085-4b91-91af-cdffe80186c0" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361768    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc2ff918-fa0b-4719-b4f1-ed8e723c96fc" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361775    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="86096906-704d-455f-aab8-4b852fb2bba6" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361782    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c572c65-36af-43d5-98e1-7cf57034a71f" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361788    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="45aef524-0f0d-4db9-aece-46db1d4e6e2f" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361796    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ff2b1336-e825-47bd-b58a-a72c71d446fb" containerName="istio-proxy"
Nov 03 00:41:15.362105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.361802    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="60317f80-10cb-4053-bfec-17853dc9fada" containerName="istio-proxy"
Nov 03 00:41:15.369477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.369453    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:15.392925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.392899    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7615-5304f0"]
Nov 03 00:41:15.400557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.400533    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5fmb\""
Nov 03 00:41:15.433363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.433232    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jncp\" (UniqueName: \"kubernetes.io/projected/16a5f5bd-bacc-49c1-a043-1f484f902701-kube-api-access-9jncp\") pod \"test-curl-7615-5304f0\" (UID: \"16a5f5bd-bacc-49c1-a043-1f484f902701\") " pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:15.535158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.535126    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jncp\" (UniqueName: \"kubernetes.io/projected/16a5f5bd-bacc-49c1-a043-1f484f902701-kube-api-access-9jncp\") pod \"test-curl-7615-5304f0\" (UID: \"16a5f5bd-bacc-49c1-a043-1f484f902701\") " pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:15.558174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.558141    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jncp\" (UniqueName: \"kubernetes.io/projected/16a5f5bd-bacc-49c1-a043-1f484f902701-kube-api-access-9jncp\") pod \"test-curl-7615-5304f0\" (UID: \"16a5f5bd-bacc-49c1-a043-1f484f902701\") " pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:15.685829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:15.685749    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:16.006800 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:16.006768    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7615-5304f0"]
Nov 03 00:41:16.007042 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:16.007018    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod16a5f5bd_bacc_49c1_a043_1f484f902701.slice/crio-f4b051c22bb2d43683f0c3ccf3b760501abae0df4a20f35a0ba79edcb854e68e WatchSource:0}: Error finding container f4b051c22bb2d43683f0c3ccf3b760501abae0df4a20f35a0ba79edcb854e68e: Status 404 returned error can't find the container with id f4b051c22bb2d43683f0c3ccf3b760501abae0df4a20f35a0ba79edcb854e68e
Nov 03 00:41:16.147051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:16.147017    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7615-5304f0" event={"ID":"16a5f5bd-bacc-49c1-a043-1f484f902701","Type":"ContainerStarted","Data":"d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9"}
Nov 03 00:41:16.147051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:16.147060    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7615-5304f0" event={"ID":"16a5f5bd-bacc-49c1-a043-1f484f902701","Type":"ContainerStarted","Data":"f4b051c22bb2d43683f0c3ccf3b760501abae0df4a20f35a0ba79edcb854e68e"}
Nov 03 00:41:16.182785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:16.182729    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7615-5304f0" podStartSLOduration=1.182711992 podStartE2EDuration="1.182711992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:16.180864789 +0000 UTC m=+2113.736216825" watchObservedRunningTime="2026-11-03 00:41:16.182711992 +0000 UTC m=+2113.738064017"
Nov 03 00:41:17.035882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:17.035848    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7615-5304f0"]
Nov 03 00:41:17.040972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:17.040644    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7615-5304f0"]
Nov 03 00:41:18.160151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:18.160114    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7615-5304f0" podUID="16a5f5bd-bacc-49c1-a043-1f484f902701" containerName="test-curl-7615-5304f0" containerID="cri-o://d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9" gracePeriod=2
Nov 03 00:41:18.172900 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:18.172867    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7615-5304f0\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="16a5f5bd-bacc-49c1-a043-1f484f902701" pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:19.044066 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.044031    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"]
Nov 03 00:41:19.045238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.045212    6000 cpu_manager.go:414] "containerMap: removing container" podUID="16a5f5bd-bacc-49c1-a043-1f484f902701" containerName="test-curl-7615-5304f0"
Nov 03 00:41:19.045382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.045241    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a5f5bd-bacc-49c1-a043-1f484f902701" containerName="test-curl-7615-5304f0"
Nov 03 00:41:19.045495 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.045478    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a5f5bd-bacc-49c1-a043-1f484f902701" containerName="test-curl-7615-5304f0"
Nov 03 00:41:19.050111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.050086    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.065413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.065388    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-d87781-openshift-default-dockercfg-f5xfs\""
Nov 03 00:41:19.081176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.081126    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"]
Nov 03 00:41:19.094488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094452    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-token\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094490    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-credential-socket\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094550    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l8fh\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-kube-api-access-8l8fh\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094572    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istiod-ca-cert\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094603    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-data\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094644    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-certs\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094759    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-podinfo\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094798    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-envoy\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.094936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.094824    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-socket\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211470    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-token\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211525    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-credential-socket\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211612    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l8fh\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-kube-api-access-8l8fh\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211643    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istiod-ca-cert\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211681    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-data\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211730    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-certs\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211900    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-podinfo\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.211972    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-envoy\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.224129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.212008    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-socket\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.230580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.227693    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-socket\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.230580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.228568    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istiod-ca-cert\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.230580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.228697    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-certs\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.230580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.228845    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-data\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.230580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.228965    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-credential-socket\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.254280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.250888    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-podinfo\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.270278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.268619    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-envoy\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.270278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.270106    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l8fh\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-kube-api-access-8l8fh\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.297352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.277523    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-token\") pod \"e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") " pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.374554 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.372523    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:19.698227 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:19.698197    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod587e6577_0386_4e2c_85e2_8e91b40c5fbc.slice/crio-7f5ec2280a65857dd05fa37b63f347e12bf2812b303760653d2ac368d8021865 WatchSource:0}: Error finding container 7f5ec2280a65857dd05fa37b63f347e12bf2812b303760653d2ac368d8021865: Status 404 returned error can't find the container with id 7f5ec2280a65857dd05fa37b63f347e12bf2812b303760653d2ac368d8021865
Nov 03 00:41:19.698552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.698529    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"]
Nov 03 00:41:19.701673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.701642    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:19.701780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.701727    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:19.701780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:19.701757    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:19.897807 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:19.897771    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:41:20.211312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.210448    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" event={"ID":"587e6577-0386-4e2c-85e2-8e91b40c5fbc","Type":"ContainerStarted","Data":"206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f"}
Nov 03 00:41:20.211312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.210513    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" event={"ID":"587e6577-0386-4e2c-85e2-8e91b40c5fbc","Type":"ContainerStarted","Data":"7f5ec2280a65857dd05fa37b63f347e12bf2812b303760653d2ac368d8021865"}
Nov 03 00:41:20.373963 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.373928    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:20.379538 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.379513    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:20.444337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.443393    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podStartSLOduration=1.443372121 podStartE2EDuration="1.443372121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:20.255612535 +0000 UTC m=+2117.810964560" watchObservedRunningTime="2026-11-03 00:41:20.443372121 +0000 UTC m=+2117.998724146"
Nov 03 00:41:20.491545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.491520    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:20.662661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.662632    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a5f5bd-bacc-49c1-a043-1f484f902701-kube-api-access-9jncp\" (UniqueName: \"kubernetes.io/projected/16a5f5bd-bacc-49c1-a043-1f484f902701-kube-api-access-9jncp\") pod \"16a5f5bd-bacc-49c1-a043-1f484f902701\" (UID: \"16a5f5bd-bacc-49c1-a043-1f484f902701\") "
Nov 03 00:41:20.665144 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.665101    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a5f5bd-bacc-49c1-a043-1f484f902701-kube-api-access-9jncp" pod "16a5f5bd-bacc-49c1-a043-1f484f902701" (UID: "16a5f5bd-bacc-49c1-a043-1f484f902701"). InnerVolumeSpecName "kube-api-access-9jncp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:20.764626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:20.764549    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jncp\" (UniqueName: \"kubernetes.io/projected/16a5f5bd-bacc-49c1-a043-1f484f902701-kube-api-access-9jncp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:21.218930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:21.218889    6000 generic.go:356] "Generic (PLEG): container finished" podID="16a5f5bd-bacc-49c1-a043-1f484f902701" containerID="d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9" exitCode=137
Nov 03 00:41:21.219150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:21.218938    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7615-5304f0"
Nov 03 00:41:21.219150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:21.218988    6000 scope.go:122] "RemoveContainer" containerID="d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9"
Nov 03 00:41:21.219343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:21.219329    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:21.220481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:21.220465    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:41:21.230192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:21.230174    6000 scope.go:122] "RemoveContainer" containerID="d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9"
Nov 03 00:41:21.230471 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:21.230452    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9\": container with ID starting with d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9 not found: ID does not exist" containerID="d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9"
Nov 03 00:41:21.230524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:21.230477    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9"} err="failed to get container status \"d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9\": rpc error: code = NotFound desc = could not find container \"d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9\": container with ID starting with d8f05b3ac26da598941afef88d291b20d197f3dc25b2287becda3eee6715a0b9 not found: ID does not exist"
Nov 03 00:41:22.474003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.473970    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"]
Nov 03 00:41:22.479511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.479484    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.503549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.503516    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"]
Nov 03 00:41:22.512915 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.508463    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-5c109b-openshift-default-dockercfg-lfstj\""
Nov 03 00:41:22.569438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.569409    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a5f5bd-bacc-49c1-a043-1f484f902701" path="/var/lib/kubelet/pods/16a5f5bd-bacc-49c1-a043-1f484f902701/volumes"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.601541    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-podinfo\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.601600    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b40ec4-3b82-4726-a95b-f80577c4b051-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.601686    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-certs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.601722    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.601814    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-credential-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.601841    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-token\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.601893    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx9vs\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-kube-api-access-rx9vs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.602036    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-envoy\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.604284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.602104    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-data\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.703871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.703831    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.703912    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-credential-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.703934    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-token\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.703968    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx9vs\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-kube-api-access-rx9vs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704120    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-envoy\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704193    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-data\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704288    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-podinfo\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704316    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b40ec4-3b82-4726-a95b-f80577c4b051-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704393    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704402    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-certs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704419    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-credential-socket\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704642    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-data\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.704999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.704979    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-certs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.705213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.705197    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b40ec4-3b82-4726-a95b-f80577c4b051-istiod-ca-cert\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.706893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.706862    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-envoy\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.707364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.707347    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-podinfo\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.734396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.734304    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-token\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.734549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.734472    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx9vs\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-kube-api-access-rx9vs\") pod \"e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") " pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:22.803603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:22.803567    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:23.130056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.130024    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"]
Nov 03 00:41:23.130508 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:23.130486    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14b40ec4_3b82_4726_a95b_f80577c4b051.slice/crio-37b21c80b0d69f63725bf9144738bbb4621dc2dbf7e29076a26075b11ff7f1e2 WatchSource:0}: Error finding container 37b21c80b0d69f63725bf9144738bbb4621dc2dbf7e29076a26075b11ff7f1e2: Status 404 returned error can't find the container with id 37b21c80b0d69f63725bf9144738bbb4621dc2dbf7e29076a26075b11ff7f1e2
Nov 03 00:41:23.133303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.133243    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:23.133379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.133333    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:23.133379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.133363    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:23.246580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.246540    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" event={"ID":"14b40ec4-3b82-4726-a95b-f80577c4b051","Type":"ContainerStarted","Data":"61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6"}
Nov 03 00:41:23.246778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.246641    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" event={"ID":"14b40ec4-3b82-4726-a95b-f80577c4b051","Type":"ContainerStarted","Data":"37b21c80b0d69f63725bf9144738bbb4621dc2dbf7e29076a26075b11ff7f1e2"}
Nov 03 00:41:23.299983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.299930    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podStartSLOduration=1.299914626 podStartE2EDuration="1.299914626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:23.297554304 +0000 UTC m=+2120.852906339" watchObservedRunningTime="2026-11-03 00:41:23.299914626 +0000 UTC m=+2120.855266649"
Nov 03 00:41:23.803708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:23.803679    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:24.419745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.419708    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"]
Nov 03 00:41:24.424856 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.424834    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.444404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.444371    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"]
Nov 03 00:41:24.451656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.451620    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-d87781\""
Nov 03 00:41:24.451797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.451660    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-d87781-dockercfg-kjt9v\""
Nov 03 00:41:24.477885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.477810    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"]
Nov 03 00:41:24.482529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.482505    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:24.504060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.504033    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"]
Nov 03 00:41:24.566283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.566217    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c90679-b97d-4691-9fa4-352d916678dd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.583663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.583612    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-462n9\" (UniqueName: \"kubernetes.io/projected/e8c90679-b97d-4691-9fa4-352d916678dd-kube-api-access-462n9\") pod \"payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.695403 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.695366    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c90679-b97d-4691-9fa4-352d916678dd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.696611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.696579    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-462n9\" (UniqueName: \"kubernetes.io/projected/e8c90679-b97d-4691-9fa4-352d916678dd-kube-api-access-462n9\") pod \"payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.698377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.698337    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:24.698660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.698626    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprmn\" (UniqueName: \"kubernetes.io/projected/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-kube-api-access-wprmn\") pod \"payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:24.701718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.696602    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c90679-b97d-4691-9fa4-352d916678dd-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.715775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.715609    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-462n9\" (UniqueName: \"kubernetes.io/projected/e8c90679-b97d-4691-9fa4-352d916678dd-kube-api-access-462n9\") pod \"payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.741330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.741240    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:24.804339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.803222    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:24.804339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.803352    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprmn\" (UniqueName: \"kubernetes.io/projected/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-kube-api-access-wprmn\") pod \"payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:24.804339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.804304    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:24.810967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.810839    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:24.827319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:24.825657    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprmn\" (UniqueName: \"kubernetes.io/projected/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-kube-api-access-wprmn\") pod \"payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:25.049830 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:25.049799    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8c90679_b97d_4691_9fa4_352d916678dd.slice/crio-e86b418ee03ff982fcb08350c0dcab01dc18f017b11fe246d6cf4a23731b362d WatchSource:0}: Error finding container e86b418ee03ff982fcb08350c0dcab01dc18f017b11fe246d6cf4a23731b362d: Status 404 returned error can't find the container with id e86b418ee03ff982fcb08350c0dcab01dc18f017b11fe246d6cf4a23731b362d
Nov 03 00:41:25.049974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.049952    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"]
Nov 03 00:41:25.103351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.103317    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:25.264991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.264956    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" event={"ID":"e8c90679-b97d-4691-9fa4-352d916678dd","Type":"ContainerStarted","Data":"c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6"}
Nov 03 00:41:25.265201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.265013    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" event={"ID":"e8c90679-b97d-4691-9fa4-352d916678dd","Type":"ContainerStarted","Data":"e86b418ee03ff982fcb08350c0dcab01dc18f017b11fe246d6cf4a23731b362d"}
Nov 03 00:41:25.265324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.265299    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:25.265376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.265349    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:25.266432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.266409    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:41:25.314935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.314806    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" podStartSLOduration=1.3147901370000001 podStartE2EDuration="1.314790137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:25.314184381 +0000 UTC m=+2122.869536405" watchObservedRunningTime="2026-11-03 00:41:25.314790137 +0000 UTC m=+2122.870142160"
Nov 03 00:41:25.417905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:25.417872    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"]
Nov 03 00:41:25.429983 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:25.429947    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d2f8092_9d61_4b06_a9d4_a304c85d9f8b.slice/crio-4bface825c4b62b89fc7168235630a996b17e1194ade1b08a33c32180a9bfeb7 WatchSource:0}: Error finding container 4bface825c4b62b89fc7168235630a996b17e1194ade1b08a33c32180a9bfeb7: Status 404 returned error can't find the container with id 4bface825c4b62b89fc7168235630a996b17e1194ade1b08a33c32180a9bfeb7
Nov 03 00:41:26.274221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:26.274184    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" event={"ID":"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b","Type":"ContainerStarted","Data":"ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b"}
Nov 03 00:41:26.274633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:26.274243    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" event={"ID":"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b","Type":"ContainerStarted","Data":"4bface825c4b62b89fc7168235630a996b17e1194ade1b08a33c32180a9bfeb7"}
Nov 03 00:41:26.274934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:26.274902    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:26.341292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:26.337455    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" podStartSLOduration=2.337434275 podStartE2EDuration="2.337434275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:26.316168433 +0000 UTC m=+2123.871520457" watchObservedRunningTime="2026-11-03 00:41:26.337434275 +0000 UTC m=+2123.892786299"
Nov 03 00:41:27.695337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.695299    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"]
Nov 03 00:41:27.700004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.699982    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:27.719908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.719869    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"]
Nov 03 00:41:27.727078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.727052    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-5c109b\""
Nov 03 00:41:27.727391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.727353    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-5c109b-dockercfg-q6jnm\""
Nov 03 00:41:27.851657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.851620    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmn2r\" (UniqueName: \"kubernetes.io/projected/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-kube-api-access-jmn2r\") pod \"payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:27.886280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.879537    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:27.886280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.879419    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"]
Nov 03 00:41:27.939680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.939635    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"]
Nov 03 00:41:27.940077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.940050    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:27.981472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.981434    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md2sw\" (UniqueName: \"kubernetes.io/projected/1e39d791-541f-4cc3-baec-5635b01020a6-kube-api-access-md2sw\") pod \"payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:27.981658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.981506    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e39d791-541f-4cc3-baec-5635b01020a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:27.981716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.981676    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmn2r\" (UniqueName: \"kubernetes.io/projected/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-kube-api-access-jmn2r\") pod \"payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:27.981754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.981718    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:27.982552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:27.982529    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:28.000095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.000063    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmn2r\" (UniqueName: \"kubernetes.io/projected/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-kube-api-access-jmn2r\") pod \"payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:28.015119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.015097    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:28.083836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.083802    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md2sw\" (UniqueName: \"kubernetes.io/projected/1e39d791-541f-4cc3-baec-5635b01020a6-kube-api-access-md2sw\") pod \"payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:28.083966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.083886    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e39d791-541f-4cc3-baec-5635b01020a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:28.084926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.084900    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e39d791-541f-4cc3-baec-5635b01020a6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:28.103753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.103723    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md2sw\" (UniqueName: \"kubernetes.io/projected/1e39d791-541f-4cc3-baec-5635b01020a6-kube-api-access-md2sw\") pod \"payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:28.268767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.268691    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:28.316194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.316166    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"]
Nov 03 00:41:28.316602 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:28.316567    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaac0b46_9a05_41c3_8c6a_cf9097d5cf37.slice/crio-6e0b7940a4ac7bc47e14c1137ce877a755422144732945e740e29c6dbc9f990f WatchSource:0}: Error finding container 6e0b7940a4ac7bc47e14c1137ce877a755422144732945e740e29c6dbc9f990f: Status 404 returned error can't find the container with id 6e0b7940a4ac7bc47e14c1137ce877a755422144732945e740e29c6dbc9f990f
Nov 03 00:41:28.580139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:28.580104    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"]
Nov 03 00:41:28.580645 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:28.580623    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e39d791_541f_4cc3_baec_5635b01020a6.slice/crio-af931b761454c336eaf649c43970a50bcb560137900191c5bb7493f1a407dc18 WatchSource:0}: Error finding container af931b761454c336eaf649c43970a50bcb560137900191c5bb7493f1a407dc18: Status 404 returned error can't find the container with id af931b761454c336eaf649c43970a50bcb560137900191c5bb7493f1a407dc18
Nov 03 00:41:29.331996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.331948    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" event={"ID":"caac0b46-9a05-41c3-8c6a-cf9097d5cf37","Type":"ContainerStarted","Data":"0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7"}
Nov 03 00:41:29.332522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.332043    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" event={"ID":"caac0b46-9a05-41c3-8c6a-cf9097d5cf37","Type":"ContainerStarted","Data":"6e0b7940a4ac7bc47e14c1137ce877a755422144732945e740e29c6dbc9f990f"}
Nov 03 00:41:29.332522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.332196    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:29.333732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.333708    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" event={"ID":"1e39d791-541f-4cc3-baec-5635b01020a6","Type":"ContainerStarted","Data":"8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd"}
Nov 03 00:41:29.333881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.333745    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" event={"ID":"1e39d791-541f-4cc3-baec-5635b01020a6","Type":"ContainerStarted","Data":"af931b761454c336eaf649c43970a50bcb560137900191c5bb7493f1a407dc18"}
Nov 03 00:41:29.333881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.333850    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:29.369327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.369281    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" podStartSLOduration=2.369247768 podStartE2EDuration="2.369247768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:29.367763252 +0000 UTC m=+2126.923115289" watchObservedRunningTime="2026-11-03 00:41:29.369247768 +0000 UTC m=+2126.924599792"
Nov 03 00:41:29.404025 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.403922    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" podStartSLOduration=2.403907336 podStartE2EDuration="2.403907336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:29.403418838 +0000 UTC m=+2126.958770875" watchObservedRunningTime="2026-11-03 00:41:29.403907336 +0000 UTC m=+2126.959259361"
Nov 03 00:41:29.598903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.598818    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7618-aa49f0"]
Nov 03 00:41:29.603288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.603273    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:29.617467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.617448    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5fmb\""
Nov 03 00:41:29.619471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.619450    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7618-aa49f0"]
Nov 03 00:41:29.708405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.708361    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tws22\" (UniqueName: \"kubernetes.io/projected/864aa400-5406-4e7e-a1fd-26ca701e1157-kube-api-access-tws22\") pod \"test-curl-7618-aa49f0\" (UID: \"864aa400-5406-4e7e-a1fd-26ca701e1157\") " pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:29.809757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.809726    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tws22\" (UniqueName: \"kubernetes.io/projected/864aa400-5406-4e7e-a1fd-26ca701e1157-kube-api-access-tws22\") pod \"test-curl-7618-aa49f0\" (UID: \"864aa400-5406-4e7e-a1fd-26ca701e1157\") " pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:29.828143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.828118    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tws22\" (UniqueName: \"kubernetes.io/projected/864aa400-5406-4e7e-a1fd-26ca701e1157-kube-api-access-tws22\") pod \"test-curl-7618-aa49f0\" (UID: \"864aa400-5406-4e7e-a1fd-26ca701e1157\") " pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:29.917330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:29.917303    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:30.216581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:30.216547    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7618-aa49f0"]
Nov 03 00:41:30.216922 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:30.216899    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod864aa400_5406_4e7e_a1fd_26ca701e1157.slice/crio-593bc7645963ef21fe2011b2567bad75451377c0564cc6b6de83e9050f8d739b WatchSource:0}: Error finding container 593bc7645963ef21fe2011b2567bad75451377c0564cc6b6de83e9050f8d739b: Status 404 returned error can't find the container with id 593bc7645963ef21fe2011b2567bad75451377c0564cc6b6de83e9050f8d739b
Nov 03 00:41:30.344153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:30.344118    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7618-aa49f0" event={"ID":"864aa400-5406-4e7e-a1fd-26ca701e1157","Type":"ContainerStarted","Data":"20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85"}
Nov 03 00:41:30.344597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:30.344161    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7618-aa49f0" event={"ID":"864aa400-5406-4e7e-a1fd-26ca701e1157","Type":"ContainerStarted","Data":"593bc7645963ef21fe2011b2567bad75451377c0564cc6b6de83e9050f8d739b"}
Nov 03 00:41:30.381639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:30.381567    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7618-aa49f0" podStartSLOduration=1.381552941 podStartE2EDuration="1.381552941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:30.378639029 +0000 UTC m=+2127.933991052" watchObservedRunningTime="2026-11-03 00:41:30.381552941 +0000 UTC m=+2127.936904964"
Nov 03 00:41:31.242955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:31.242921    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7618-aa49f0"]
Nov 03 00:41:31.247439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:31.247410    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7618-aa49f0"]
Nov 03 00:41:32.358586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:32.358537    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7618-aa49f0" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" containerName="test-curl-7618-aa49f0" containerID="cri-o://20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85" gracePeriod=2
Nov 03 00:41:32.371476 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:32.371441    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7618-aa49f0\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:32.575639 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:32.575605    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7618-aa49f0\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:34.574314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:34.574286    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:34.685241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:34.685144    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864aa400-5406-4e7e-a1fd-26ca701e1157-kube-api-access-tws22\" (UniqueName: \"kubernetes.io/projected/864aa400-5406-4e7e-a1fd-26ca701e1157-kube-api-access-tws22\") pod \"864aa400-5406-4e7e-a1fd-26ca701e1157\" (UID: \"864aa400-5406-4e7e-a1fd-26ca701e1157\") "
Nov 03 00:41:34.687535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:34.687509    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864aa400-5406-4e7e-a1fd-26ca701e1157-kube-api-access-tws22" pod "864aa400-5406-4e7e-a1fd-26ca701e1157" (UID: "864aa400-5406-4e7e-a1fd-26ca701e1157"). InnerVolumeSpecName "kube-api-access-tws22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:34.787437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:34.787406    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tws22\" (UniqueName: \"kubernetes.io/projected/864aa400-5406-4e7e-a1fd-26ca701e1157-kube-api-access-tws22\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:35.380528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:35.380486    6000 generic.go:356] "Generic (PLEG): container finished" podID="864aa400-5406-4e7e-a1fd-26ca701e1157" containerID="20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85" exitCode=137
Nov 03 00:41:35.380721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:35.380541    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:35.380721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:35.380561    6000 scope.go:122] "RemoveContainer" containerID="20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85"
Nov 03 00:41:35.391772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:35.391756    6000 scope.go:122] "RemoveContainer" containerID="20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85"
Nov 03 00:41:35.392072 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:35.392045    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85\": container with ID starting with 20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85 not found: ID does not exist" containerID="20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85"
Nov 03 00:41:35.392134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:35.392082    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85"} err="failed to get container status \"20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85\": rpc error: code = NotFound desc = could not find container \"20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85\": container with ID starting with 20c6cea3aaf35332ab6bb39c6dd0392e110b537020218817bde3f592a75bec85 not found: ID does not exist"
Nov 03 00:41:35.401723 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:35.401699    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7618-aa49f0\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:36.278017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.277987    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 00:41:36.290695 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:36.290666    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7618-aa49f0\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" pod="opendatahub/test-curl-7618-aa49f0"
Nov 03 00:41:36.569935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.569857    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" path="/var/lib/kubelet/pods/864aa400-5406-4e7e-a1fd-26ca701e1157/volumes"
Nov 03 00:41:36.581613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.581583    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"]
Nov 03 00:41:36.582363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.582348    6000 cpu_manager.go:414] "containerMap: removing container" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" containerName="test-curl-7618-aa49f0"
Nov 03 00:41:36.582435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.582364    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" containerName="test-curl-7618-aa49f0"
Nov 03 00:41:36.582482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.582472    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="864aa400-5406-4e7e-a1fd-26ca701e1157" containerName="test-curl-7618-aa49f0"
Nov 03 00:41:36.587875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.587859    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.611674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.611640    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"]
Nov 03 00:41:36.696345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.696312    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-099583-openshift-default-dockercfg-hbtbr\""
Nov 03 00:41:36.723326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723289    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723399    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mst2c\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-kube-api-access-mst2c\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723433    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5d410-6702-46d1-a442-6d712cd0705f-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723460    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723536    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723593    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723640    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723681    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5d410-6702-46d1-a442-6d712cd0705f-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.723732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.723713    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825006    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825095    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825121    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825153    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5d410-6702-46d1-a442-6d712cd0705f-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825197    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825379    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825489    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mst2c\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-kube-api-access-mst2c\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825524    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825541    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5d410-6702-46d1-a442-6d712cd0705f-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825602    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825686    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825683    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.825884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.825825    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.826185 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.826154    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5d410-6702-46d1-a442-6d712cd0705f-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.827919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.827896    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.828017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.827954    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5d410-6702-46d1-a442-6d712cd0705f-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.844120 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.844092    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mst2c\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-kube-api-access-mst2c\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.844326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.844306    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:36.912245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:36.912219    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:37.228893 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:41:37.228856    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68d5d410_6702_46d1_a442_6d712cd0705f.slice/crio-1b46031adaf42707fbae98ff42bba8bad8a8dc4e98c3820c75010da6a71fd2ea WatchSource:0}: Error finding container 1b46031adaf42707fbae98ff42bba8bad8a8dc4e98c3820c75010da6a71fd2ea: Status 404 returned error can't find the container with id 1b46031adaf42707fbae98ff42bba8bad8a8dc4e98c3820c75010da6a71fd2ea
Nov 03 00:41:37.229060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.229038    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"]
Nov 03 00:41:37.231668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.231638    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:37.231764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.231706    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:37.231764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.231735    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:41:37.400006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.399969    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" event={"ID":"68d5d410-6702-46d1-a442-6d712cd0705f","Type":"ContainerStarted","Data":"28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647"}
Nov 03 00:41:37.400529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.400028    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" event={"ID":"68d5d410-6702-46d1-a442-6d712cd0705f","Type":"ContainerStarted","Data":"1b46031adaf42707fbae98ff42bba8bad8a8dc4e98c3820c75010da6a71fd2ea"}
Nov 03 00:41:37.448599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.448537    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" podStartSLOduration=1.448518806 podStartE2EDuration="1.448518806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:41:37.445443368 +0000 UTC m=+2135.000795415" watchObservedRunningTime="2026-11-03 00:41:37.448518806 +0000 UTC m=+2135.003870833"
Nov 03 00:41:37.912372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:37.912332    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:38.326695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:38.326604    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 00:41:38.915105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:38.915057    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 00:41:38.915618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:38.915127    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 00:41:39.912903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:39.912858    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 00:41:39.913108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:39.912933    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 00:41:40.347676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:40.347580    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 00:41:40.348103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:40.347781    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 00:41:40.912730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:40.912695    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 00:41:40.912929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:40.912760    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 00:41:41.913477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:41.913441    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 00:41:41.913993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:41.913547    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 00:41:42.918290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:42.914631    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused" start-of-body=
Nov 03 00:41:42.918290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:42.914696    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.182:15021/healthz/ready\": dial tcp 10.134.0.182:15021: connect: connection refused"
Nov 03 00:41:43.917629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:43.917593    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:43.918164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:43.918136    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:43.918845 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:43.918821    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:41:49.913927 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:49.913882    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:41:52.289046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:52.289000    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"]
Nov 03 00:41:52.289724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:52.289454    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" containerID="cri-o://7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548" gracePeriod=30
Nov 03 00:41:54.271129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:54.271092    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"]
Nov 03 00:41:54.878516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:54.878482    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:41:56.738151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:56.738113    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:41:56.738818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:56.738784    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:41:57.593857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.593775    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:41:57.675601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.675563    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-data\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.675799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.675608    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-kube-api-access-6nsd2\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-kube-api-access-6nsd2\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.675799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.675641    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istiod-ca-cert\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.675799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.675695    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-certs\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.675799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.675731    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-credential-socket\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.676038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.675818    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-socket\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.676038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.675849    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-envoy\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.676953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.676914    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istiod-ca-cert" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:41:57.677115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.677047    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-socket" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.677183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.677136    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-data" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.677472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.677449    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-certs" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.677595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.677564    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-credential-socket" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.679189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.679152    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-envoy" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:41:57.679779 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.679755    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-kube-api-access-6nsd2" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "kube-api-access-6nsd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.697974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.697948    6000 generic.go:356] "Generic (PLEG): container finished" podID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerID="7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548" exitCode=0
Nov 03 00:41:57.698153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.698133    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" event={"ID":"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5","Type":"ContainerDied","Data":"7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548"}
Nov 03 00:41:57.698239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.698206    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg" event={"ID":"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5","Type":"ContainerDied","Data":"5f6a8e4b5698254ace44022a5c1626f1ccce2fb89ad88b9fa4af7a4172995612"}
Nov 03 00:41:57.698239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.698232    6000 scope.go:122] "RemoveContainer" containerID="7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548"
Nov 03 00:41:57.698421 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.698406    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"
Nov 03 00:41:57.723469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.723409    6000 scope.go:122] "RemoveContainer" containerID="7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548"
Nov 03 00:41:57.723804 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:41:57.723780    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548\": container with ID starting with 7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548 not found: ID does not exist" containerID="7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548"
Nov 03 00:41:57.723902 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.723815    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548"} err="failed to get container status \"7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548\": rpc error: code = NotFound desc = could not find container \"7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548\": container with ID starting with 7b0d7cf252b20f8615d481e668d6b9612ac26341a5a7dbd47fbd7d60d1595548 not found: ID does not exist"
Nov 03 00:41:57.777054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777004    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-podinfo\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.777614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777065    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-token\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-token\") pod \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\" (UID: \"402e4e66-b1a9-448b-94b5-1da2ff1ca5a5\") "
Nov 03 00:41:57.777763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777653    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.777763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777669    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.777763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777677    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.777763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777686    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nsd2\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-kube-api-access-6nsd2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.777763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777696    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.777763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777704    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.777763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.777712    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.779857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.779828    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-podinfo" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:41:57.779964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.779944    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-token" pod "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" (UID: "402e4e66-b1a9-448b-94b5-1da2ff1ca5a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:41:57.879217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.879181    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:57.879217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:57.879220    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:41:58.110371 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:58.108395    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"]
Nov 03 00:41:58.114357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:58.114097    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-8x5pg"]
Nov 03 00:41:58.570532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:41:58.570498    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" path="/var/lib/kubelet/pods/402e4e66-b1a9-448b-94b5-1da2ff1ca5a5/volumes"
Nov 03 00:42:03.918070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:03.918036    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"]
Nov 03 00:42:09.295224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.295160    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7" podUID="5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" containerName="main" containerID="cri-o://7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b" gracePeriod=45
Nov 03 00:42:09.587795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.587766    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:42:09.692209 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692177    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-dshm\") pod \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") "
Nov 03 00:42:09.692406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692329    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-model-cache\") pod \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") "
Nov 03 00:42:09.692406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692349    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tmp-dir\") pod \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") "
Nov 03 00:42:09.692503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692427    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-home\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-home\") pod \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") "
Nov 03 00:42:09.692503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692454    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tls-certs\") pod \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") "
Nov 03 00:42:09.692623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692501    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-kube-api-access-fkq6r\" (UniqueName: \"kubernetes.io/projected/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-kube-api-access-fkq6r\") pod \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\" (UID: \"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b\") "
Nov 03 00:42:09.692680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692622    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-model-cache" pod "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" (UID: "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.692764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692731    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-home" pod "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" (UID: "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.692764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.692747    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tmp-dir" pod "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" (UID: "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.693269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.693211    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.693269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.693235    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.693512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.693248    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.694728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.694702    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-dshm" pod "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" (UID: "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:09.694824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.694754    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tls-certs" pod "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" (UID: "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:09.695016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.694995    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-kube-api-access-fkq6r" pod "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" (UID: "5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b"). InnerVolumeSpecName "kube-api-access-fkq6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:09.794271 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794234    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.794271 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794276    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.794489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794286    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkq6r\" (UniqueName: \"kubernetes.io/projected/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b-kube-api-access-fkq6r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:09.794773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794741    6000 generic.go:356] "Generic (PLEG): container finished" podID="5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" containerID="7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b" exitCode=0
Nov 03 00:42:09.794905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794798    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"
Nov 03 00:42:09.794905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794832    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7" event={"ID":"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b","Type":"ContainerDied","Data":"7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b"}
Nov 03 00:42:09.795102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794933    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7" event={"ID":"5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b","Type":"ContainerDied","Data":"37038224bbec1ebdbdeaf7dd5113c8d2c197668edf7cd9592d08443a07f9242a"}
Nov 03 00:42:09.795102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.794957    6000 scope.go:122] "RemoveContainer" containerID="7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b"
Nov 03 00:42:09.806665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.806645    6000 scope.go:122] "RemoveContainer" containerID="7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b"
Nov 03 00:42:09.806964 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:42:09.806935    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b\": container with ID starting with 7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b not found: ID does not exist" containerID="7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b"
Nov 03 00:42:09.806964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.806972    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b"} err="failed to get container status \"7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b\": rpc error: code = NotFound desc = could not find container \"7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b\": container with ID starting with 7d85ac2e590a0d028c550c950e68f0e6bb0df7b33a3010acf4d2d30fe54ad52b not found: ID does not exist"
Nov 03 00:42:09.846015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.845971    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"]
Nov 03 00:42:09.852903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:09.852880    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/auth-test-model-9908e8-kserve-7f544b64b9-sfwm7"]
Nov 03 00:42:10.569429 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:10.569397    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" path="/var/lib/kubelet/pods/5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b/volumes"
Nov 03 00:42:18.941702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:18.941649    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd" podUID="46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" containerName="main" containerID="cri-o://70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3" gracePeriod=45
Nov 03 00:42:19.280605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.280570    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:42:19.435323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435283    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-home\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-home\") pod \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") "
Nov 03 00:42:19.435520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435352    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tmp-dir\") pod \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") "
Nov 03 00:42:19.435597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435521    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tls-certs\" (UniqueName: \"kubernetes.io/secret/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tls-certs\") pod \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") "
Nov 03 00:42:19.435597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435563    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-kube-api-access-p469w\") pod \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") "
Nov 03 00:42:19.435714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435583    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-home" pod "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" (UID: "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:19.435714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435611    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-model-cache\") pod \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") "
Nov 03 00:42:19.435844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435825    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-dshm\") pod \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\" (UID: \"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc\") "
Nov 03 00:42:19.435900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435878    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-model-cache" pod "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" (UID: "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:19.435960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.435941    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tmp-dir" pod "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" (UID: "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:19.436613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.436585    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:19.436613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.436613    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:19.436813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.436627    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:19.438888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.438849    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tls-certs" pod "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" (UID: "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 00:42:19.438888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.438863    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-dshm" pod "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" (UID: "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:42:19.439157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.439130    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-kube-api-access-p469w" pod "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" (UID: "46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc"). InnerVolumeSpecName "kube-api-access-p469w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:42:19.537609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.537525    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:19.537609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.537558    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p469w\" (UniqueName: \"kubernetes.io/projected/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-kube-api-access-p469w\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:19.537609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.537569    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:42:19.909970 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:42:19.909933    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:42:19.997688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.997652    6000 generic.go:356] "Generic (PLEG): container finished" podID="46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" containerID="70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3" exitCode=0
Nov 03 00:42:19.998171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.997718    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"
Nov 03 00:42:19.998171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.997745    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd" event={"ID":"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc","Type":"ContainerDied","Data":"70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3"}
Nov 03 00:42:19.998171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.997792    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd" event={"ID":"46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc","Type":"ContainerDied","Data":"dcbd924dd2edde87a05b822b4f0d49660951246c44cf2615d5d75e294ef60be1"}
Nov 03 00:42:19.998171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:19.997812    6000 scope.go:122] "RemoveContainer" containerID="70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3"
Nov 03 00:42:20.010810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:20.010779    6000 scope.go:122] "RemoveContainer" containerID="70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3"
Nov 03 00:42:20.011174 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:42:20.011148    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3\": container with ID starting with 70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3 not found: ID does not exist" containerID="70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3"
Nov 03 00:42:20.011296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:20.011192    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3"} err="failed to get container status \"70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3\": rpc error: code = NotFound desc = could not find container \"70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3\": container with ID starting with 70e7385b0b7dfbffae10618512b8ef64b6f0f2a796bdd24e95af4ad669ddaeb3 not found: ID does not exist"
Nov 03 00:42:20.049012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:20.048976    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"]
Nov 03 00:42:20.060110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:20.060076    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd"]
Nov 03 00:42:20.164455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:20.164352    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-a07a90/auth-test-model-a07a90-kserve-7d4b4467c7-hqqnd" podUID="46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.170:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:42:20.569918 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:20.569830    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" path="/var/lib/kubelet/pods/46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc/volumes"
Nov 03 00:42:22.261423 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:22.261374    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:42:22.262012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:22.261451    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:42:23.028599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:23.028568    6000 generic.go:356] "Generic (PLEG): container finished" podID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerID="61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6" exitCode=0
Nov 03 00:42:23.030652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:23.028640    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" event={"ID":"14b40ec4-3b82-4726-a95b-f80577c4b051","Type":"ContainerDied","Data":"61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6"}
Nov 03 00:42:23.030652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:23.029073    6000 scope.go:122] "RemoveContainer" containerID="61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6"
Nov 03 00:42:23.031057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:23.030762    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:42:23.031057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:23.030825    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:42:23.031057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:23.030856    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:42:24.040668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.038324    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" event={"ID":"14b40ec4-3b82-4726-a95b-f80577c4b051","Type":"ContainerStarted","Data":"f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648"}
Nov 03 00:42:24.043118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.043092    6000 generic.go:356] "Generic (PLEG): container finished" podID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerID="206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f" exitCode=0
Nov 03 00:42:24.043288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.043167    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" event={"ID":"587e6577-0386-4e2c-85e2-8e91b40c5fbc","Type":"ContainerDied","Data":"206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f"}
Nov 03 00:42:24.043594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.043579    6000 scope.go:122] "RemoveContainer" containerID="206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f"
Nov 03 00:42:24.045308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.045282    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:42:24.045369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.045357    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:42:24.045405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.045391    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:42:24.813080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.808318    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 00:42:24.813080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.808389    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 00:42:24.904204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:24.903076    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:42:25.053003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:25.052970    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" event={"ID":"587e6577-0386-4e2c-85e2-8e91b40c5fbc","Type":"ContainerStarted","Data":"7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5"}
Nov 03 00:42:25.267578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:25.267537    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:42:25.384761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:25.384724    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:42:25.384983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:25.384860    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 03 00:42:25.384983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:25.384909    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 03 00:42:25.815281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:25.809758    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 00:42:25.815281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:25.809825    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 00:42:26.373507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:26.373475    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 03 00:42:26.373907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:26.373527    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 03 00:42:26.822338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:26.814719    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 00:42:26.822338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:26.814786    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 00:42:27.373707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:27.373675    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 03 00:42:27.374172 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:27.374148    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 03 00:42:27.806891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:27.806775    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 00:42:27.807186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:27.807143    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 00:42:28.374303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:28.374245    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 03 00:42:28.374748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:28.374320    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 03 00:42:28.811338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:28.810693    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused" start-of-body=
Nov 03 00:42:28.811338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:28.810760    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.176:15021/healthz/ready\": dial tcp 10.134.0.176:15021: connect: connection refused"
Nov 03 00:42:29.377140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:29.377100    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 03 00:42:29.377803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:29.377771    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 03 00:42:29.822900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:29.822805    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:42:29.825086 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:29.825061    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 00:42:30.373965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:30.373933    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body=
Nov 03 00:42:30.374159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:30.373984    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused"
Nov 03 00:42:31.378189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:31.378157    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:42:31.378618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:31.378454    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:42:31.379164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:31.379147    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 00:42:37.812509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:37.812466    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:42:46.280440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.280390    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"]
Nov 03 00:42:46.281987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.281964    6000 cpu_manager.go:414] "containerMap: removing container" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy"
Nov 03 00:42:46.282169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.282156    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy"
Nov 03 00:42:46.282292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.282281    6000 cpu_manager.go:414] "containerMap: removing container" podUID="5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" containerName="main"
Nov 03 00:42:46.282410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.282399    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" containerName="main"
Nov 03 00:42:46.282578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.282567    6000 cpu_manager.go:414] "containerMap: removing container" podUID="46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" containerName="main"
Nov 03 00:42:46.282658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.282648    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" containerName="main"
Nov 03 00:42:46.283424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.283394    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f3663df-f51a-46d5-bfb8-fdab8a4d6c4b" containerName="main"
Nov 03 00:42:46.283574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.283432    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="46b6faa9-61dc-4cbc-ab8d-46d19bc3cedc" containerName="main"
Nov 03 00:42:46.283574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.283460    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="402e4e66-b1a9-448b-94b5-1da2ff1ca5a5" containerName="istio-proxy"
Nov 03 00:42:46.288611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.288584    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.306747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.306716    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"]
Nov 03 00:42:46.309431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.309404    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-955dc3-serving-cert\""
Nov 03 00:42:46.384855 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.384813    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.385012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.384882    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8sv6\" (UniqueName: \"kubernetes.io/projected/4729ad2b-ba3f-47f0-9cae-7c843a10781f-kube-api-access-f8sv6\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.385012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.384960    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-metrics-certs\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.385012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.385004    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729ad2b-ba3f-47f0-9cae-7c843a10781f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.486220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.486185    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8sv6\" (UniqueName: \"kubernetes.io/projected/4729ad2b-ba3f-47f0-9cae-7c843a10781f-kube-api-access-f8sv6\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.486457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.486246    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-metrics-certs\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.486457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.486323    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729ad2b-ba3f-47f0-9cae-7c843a10781f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.486612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.486593    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.486996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.486965    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729ad2b-ba3f-47f0-9cae-7c843a10781f-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.488934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.488912    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-metrics-certs\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.489076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.489059    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-maas-api-tls\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.504748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.504720    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8sv6\" (UniqueName: \"kubernetes.io/projected/4729ad2b-ba3f-47f0-9cae-7c843a10781f-kube-api-access-f8sv6\") pod \"maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.610082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.610054    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:46.934031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.933924    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"]
Nov 03 00:42:46.947059 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:42:46.947028    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4729ad2b_ba3f_47f0_9cae_7c843a10781f.slice/crio-fb8739c5378dc85168208a4a97f9d224a31f3a097a2b5dc402fdd8a125f5647d WatchSource:0}: Error finding container fb8739c5378dc85168208a4a97f9d224a31f3a097a2b5dc402fdd8a125f5647d: Status 404 returned error can't find the container with id fb8739c5378dc85168208a4a97f9d224a31f3a097a2b5dc402fdd8a125f5647d
Nov 03 00:42:46.948738 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:46.948715    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:42:47.350328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:47.350300    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh" event={"ID":"4729ad2b-ba3f-47f0-9cae-7c843a10781f","Type":"ContainerStarted","Data":"fb8739c5378dc85168208a4a97f9d224a31f3a097a2b5dc402fdd8a125f5647d"}
Nov 03 00:42:48.359162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:48.359128    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh" event={"ID":"4729ad2b-ba3f-47f0-9cae-7c843a10781f","Type":"ContainerStarted","Data":"ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1"}
Nov 03 00:42:48.359605 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:48.359233    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:42:48.410199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:48.410151    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh" podStartSLOduration=2.068964426 podStartE2EDuration="2.41013722s" totalImagesPullingTime="341.172794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:42:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:42:48.39543598 +0000 UTC m=+2205.950788005" watchObservedRunningTime="2026-11-03 00:42:48.41013722 +0000 UTC m=+2205.965489245"
Nov 03 00:42:49.898125 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:42:49.898080    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:42:54.371786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:42:54.371761    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 00:43:04.598280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:04.581155    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:43:16.337695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.337661    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"]
Nov 03 00:43:16.342008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.341988    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.348765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.348736    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-kube-api-access-bc225\") pod \"payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.348903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.348837    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.357600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.357387    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-850433\""
Nov 03 00:43:16.373947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.373917    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"]
Nov 03 00:43:16.374475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.374450    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-850433-dockercfg-bj5ph\""
Nov 03 00:43:16.475428 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.475390    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.475691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.475665    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-kube-api-access-bc225\") pod \"payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.476946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.476917    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.523278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.520181    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-kube-api-access-bc225\") pod \"payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.671315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.671031    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:16.997079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:16.997048    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"]
Nov 03 00:43:16.997410 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:43:16.997388    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbb9852_0c50_49c4_bb1b_631b6c394e8b.slice/crio-f15e69f67b10ed03b552b9f1507c4112132aff148c704e32d631347e977b2976 WatchSource:0}: Error finding container f15e69f67b10ed03b552b9f1507c4112132aff148c704e32d631347e977b2976: Status 404 returned error can't find the container with id f15e69f67b10ed03b552b9f1507c4112132aff148c704e32d631347e977b2976
Nov 03 00:43:17.707729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:17.707694    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" event={"ID":"4cbb9852-0c50-49c4-bb1b-631b6c394e8b","Type":"ContainerStarted","Data":"f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613"}
Nov 03 00:43:17.708137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:17.707747    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" event={"ID":"4cbb9852-0c50-49c4-bb1b-631b6c394e8b","Type":"ContainerStarted","Data":"f15e69f67b10ed03b552b9f1507c4112132aff148c704e32d631347e977b2976"}
Nov 03 00:43:17.708137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:17.707880    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:17.746678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:17.746611    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" podStartSLOduration=1.7465980490000002 podStartE2EDuration="1.746598049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:17.743939356 +0000 UTC m=+2235.299291380" watchObservedRunningTime="2026-11-03 00:43:17.746598049 +0000 UTC m=+2235.301950109"
Nov 03 00:43:19.897775 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:43:19.897734    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:43:21.522606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.522571    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"]
Nov 03 00:43:21.526441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.526421    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.559282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.559137    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-extmod-a-c738e0-serving-cert\""
Nov 03 00:43:21.560319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.560289    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-metrics-certs\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.560437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.560418    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.560480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.560470    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsnzr\" (UniqueName: \"kubernetes.io/projected/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-kube-api-access-fsnzr\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.560661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.560645    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.564694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.564669    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"]
Nov 03 00:43:21.662021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.661990    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-metrics-certs\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.662203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.662082    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.662203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.662115    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsnzr\" (UniqueName: \"kubernetes.io/projected/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-kube-api-access-fsnzr\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.662339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.662240    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.662904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.662877    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-openshift-service-ca\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.664682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.664658    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-maas-api-tls\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.664786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.664770    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-metrics-certs\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.681084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.681057    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsnzr\" (UniqueName: \"kubernetes.io/projected/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-kube-api-access-fsnzr\") pod \"maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") " pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:21.840486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:21.840405    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:22.146875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:22.146847    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"]
Nov 03 00:43:22.147161 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:43:22.147140    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fd57543_8daa_4be4_a9f9_0208bcd55b4e.slice/crio-a3f86fb87a8a3f63d1c98e33d936ae57be770fecbf832573603c7e875326b4b1 WatchSource:0}: Error finding container a3f86fb87a8a3f63d1c98e33d936ae57be770fecbf832573603c7e875326b4b1: Status 404 returned error can't find the container with id a3f86fb87a8a3f63d1c98e33d936ae57be770fecbf832573603c7e875326b4b1
Nov 03 00:43:22.828161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:22.826869    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb" event={"ID":"0fd57543-8daa-4be4-a9f9-0208bcd55b4e","Type":"ContainerStarted","Data":"ece9e01b156f379e5f90508bc7fe84cee2f2ac369a092d9e3eb077fa3d17fe21"}
Nov 03 00:43:22.828161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:22.826941    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb" event={"ID":"0fd57543-8daa-4be4-a9f9-0208bcd55b4e","Type":"ContainerStarted","Data":"a3f86fb87a8a3f63d1c98e33d936ae57be770fecbf832573603c7e875326b4b1"}
Nov 03 00:43:22.828161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:22.828055    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:22.914628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:22.914573    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb" podStartSLOduration=1.620684841 podStartE2EDuration="1.914555327s" totalImagesPullingTime="293.870486ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:22.90288508 +0000 UTC m=+2240.458237105" watchObservedRunningTime="2026-11-03 00:43:22.914555327 +0000 UTC m=+2240.469907348"
Nov 03 00:43:28.717928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:28.717900    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 00:43:29.848903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:29.848870    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 00:43:39.506697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.506658    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"]
Nov 03 00:43:39.511053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.511018    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.528337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.528312    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"]
Nov 03 00:43:39.535147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.535113    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-909268\""
Nov 03 00:43:39.594281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.591113    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcjbj\" (UniqueName: \"kubernetes.io/projected/a7811f9d-d30d-4fee-850f-10c22d42e2cc-kube-api-access-xcjbj\") pod \"payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.594281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.591237    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7811f9d-d30d-4fee-850f-10c22d42e2cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.594281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.593026    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-909268-dockercfg-k459g\""
Nov 03 00:43:39.725286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.723643    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcjbj\" (UniqueName: \"kubernetes.io/projected/a7811f9d-d30d-4fee-850f-10c22d42e2cc-kube-api-access-xcjbj\") pod \"payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.725286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.723779    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7811f9d-d30d-4fee-850f-10c22d42e2cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.725286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.725032    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7811f9d-d30d-4fee-850f-10c22d42e2cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.739348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.739305    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"]
Nov 03 00:43:39.750137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.750113    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:39.754992 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.754961    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"]
Nov 03 00:43:39.758689 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.758571    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcjbj\" (UniqueName: \"kubernetes.io/projected/a7811f9d-d30d-4fee-850f-10c22d42e2cc-kube-api-access-xcjbj\") pod \"payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.827148 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.827116    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:39.927992 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.927953    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48vvl\" (UniqueName: \"kubernetes.io/projected/78254f06-bf41-44d1-8c79-b236c12f380e-kube-api-access-48vvl\") pod \"payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:39.928204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:39.928166    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78254f06-bf41-44d1-8c79-b236c12f380e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:40.030061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.029977    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48vvl\" (UniqueName: \"kubernetes.io/projected/78254f06-bf41-44d1-8c79-b236c12f380e-kube-api-access-48vvl\") pod \"payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:40.030231 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.030084    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78254f06-bf41-44d1-8c79-b236c12f380e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:40.030812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.030792    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78254f06-bf41-44d1-8c79-b236c12f380e-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:40.049623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.049593    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48vvl\" (UniqueName: \"kubernetes.io/projected/78254f06-bf41-44d1-8c79-b236c12f380e-kube-api-access-48vvl\") pod \"payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:40.109635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.109605    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:40.128542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.128512    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"]
Nov 03 00:43:40.129789 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:43:40.129763    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7811f9d_d30d_4fee_850f_10c22d42e2cc.slice/crio-7fee264a90c52948ede5a4c9e6ef92a37ce431af935b94dd878b88c7df50d9de WatchSource:0}: Error finding container 7fee264a90c52948ede5a4c9e6ef92a37ce431af935b94dd878b88c7df50d9de: Status 404 returned error can't find the container with id 7fee264a90c52948ede5a4c9e6ef92a37ce431af935b94dd878b88c7df50d9de
Nov 03 00:43:40.413739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.413714    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"]
Nov 03 00:43:40.413871 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:43:40.413825    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78254f06_bf41_44d1_8c79_b236c12f380e.slice/crio-b8e30a724bf8af57733e22ae198c5872d0ed5d20db440308ced9c5a31a254bc7 WatchSource:0}: Error finding container b8e30a724bf8af57733e22ae198c5872d0ed5d20db440308ced9c5a31a254bc7: Status 404 returned error can't find the container with id b8e30a724bf8af57733e22ae198c5872d0ed5d20db440308ced9c5a31a254bc7
Nov 03 00:43:40.965472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.965433    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" event={"ID":"78254f06-bf41-44d1-8c79-b236c12f380e","Type":"ContainerStarted","Data":"ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e"}
Nov 03 00:43:40.965995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.965496    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" event={"ID":"78254f06-bf41-44d1-8c79-b236c12f380e","Type":"ContainerStarted","Data":"b8e30a724bf8af57733e22ae198c5872d0ed5d20db440308ced9c5a31a254bc7"}
Nov 03 00:43:40.965995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.965524    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:40.967028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.966997    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" event={"ID":"a7811f9d-d30d-4fee-850f-10c22d42e2cc","Type":"ContainerStarted","Data":"1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282"}
Nov 03 00:43:40.967574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.967052    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:43:40.967574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:40.967071    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" event={"ID":"a7811f9d-d30d-4fee-850f-10c22d42e2cc","Type":"ContainerStarted","Data":"7fee264a90c52948ede5a4c9e6ef92a37ce431af935b94dd878b88c7df50d9de"}
Nov 03 00:43:41.002343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:41.002220    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" podStartSLOduration=2.002204294 podStartE2EDuration="2.002204294s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:41.001403753 +0000 UTC m=+2258.556755778" watchObservedRunningTime="2026-11-03 00:43:41.002204294 +0000 UTC m=+2258.557556319"
Nov 03 00:43:41.040239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:41.040183    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" podStartSLOduration=2.040170268 podStartE2EDuration="2.040170268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:43:41.0370423 +0000 UTC m=+2258.592394325" watchObservedRunningTime="2026-11-03 00:43:41.040170268 +0000 UTC m=+2258.595522293"
Nov 03 00:43:47.396020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:47.395942    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:43:49.898362 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:43:49.898326    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:43:51.977402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:51.977373    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 00:43:51.977840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:43:51.977439    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 00:44:04.892908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.892871    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"]
Nov 03 00:44:04.896787 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.896769    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:04.911046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.911025    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d4d8c2\""
Nov 03 00:44:04.914907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.914880    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"]
Nov 03 00:44:04.922093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.922072    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"agc-cred-de0db39a1fbf\""
Nov 03 00:44:04.922093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.922079    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"openshift-service-ca.crt\""
Nov 03 00:44:04.922321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.922156    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"routing-overlay\""
Nov 03 00:44:04.922321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.922184    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"kube-root-ca.crt\""
Nov 03 00:44:04.922556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:04.922536    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"payload-processing-external-model-e2e--b53073b3-dockercfg-6685v\""
Nov 03 00:44:05.026774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.026329    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzj6b\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-kube-api-access-vzj6b\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.026774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.026407    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.026774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.026495    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.026774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.026523    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.129910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.129869    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.130195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.130169    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.130705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.130684    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzj6b\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-kube-api-access-vzj6b\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.130874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.130856    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.141184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.139600    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.141184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.139683    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.141976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.141948    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.158188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.158129    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzj6b\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-kube-api-access-vzj6b\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.211852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.211821    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:05.515026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:05.514953    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"]
Nov 03 00:44:05.519298 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:44:05.519225    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373bc38c_e76a_4f1d_a73a_9763d466f097.slice/crio-cc314bc896477d869aef711fd8711be7b9ced6f4979dcf8984e29ff57d2432f6 WatchSource:0}: Error finding container cc314bc896477d869aef711fd8711be7b9ced6f4979dcf8984e29ff57d2432f6: Status 404 returned error can't find the container with id cc314bc896477d869aef711fd8711be7b9ced6f4979dcf8984e29ff57d2432f6
Nov 03 00:44:06.186767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:06.186730    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c" event={"ID":"373bc38c-e76a-4f1d-a73a-9763d466f097","Type":"ContainerStarted","Data":"6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6"}
Nov 03 00:44:06.187179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:06.186786    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c" event={"ID":"373bc38c-e76a-4f1d-a73a-9763d466f097","Type":"ContainerStarted","Data":"cc314bc896477d869aef711fd8711be7b9ced6f4979dcf8984e29ff57d2432f6"}
Nov 03 00:44:06.187179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:06.186821    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:06.225180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:06.225130    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c" podStartSLOduration=2.225115547 podStartE2EDuration="2.225115547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:06.223478408 +0000 UTC m=+2283.778830431" watchObservedRunningTime="2026-11-03 00:44:06.225115547 +0000 UTC m=+2283.780467642"
Nov 03 00:44:13.929077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:13.929045    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"]
Nov 03 00:44:13.933467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:13.933451    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:13.949778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:13.949753    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-0db47e\""
Nov 03 00:44:13.961243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:13.961219    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-0db47e-dockercfg-qqhvv\""
Nov 03 00:44:13.966530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:13.966503    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"]
Nov 03 00:44:14.004610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.004581    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxt44\" (UniqueName: \"kubernetes.io/projected/c746ad54-d687-422d-b0f4-3ba0e6f31a55-kube-api-access-wxt44\") pod \"payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:14.004783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.004762    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c746ad54-d687-422d-b0f4-3ba0e6f31a55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:14.112277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.111315    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c746ad54-d687-422d-b0f4-3ba0e6f31a55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:14.112277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.111585    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxt44\" (UniqueName: \"kubernetes.io/projected/c746ad54-d687-422d-b0f4-3ba0e6f31a55-kube-api-access-wxt44\") pod \"payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:14.116353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.112894    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c746ad54-d687-422d-b0f4-3ba0e6f31a55-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:14.172628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.172593    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxt44\" (UniqueName: \"kubernetes.io/projected/c746ad54-d687-422d-b0f4-3ba0e6f31a55-kube-api-access-wxt44\") pod \"payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:14.248114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.248026    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:14.554993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:14.554960    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"]
Nov 03 00:44:14.555580 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:44:14.555553    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc746ad54_d687_422d_b0f4_3ba0e6f31a55.slice/crio-af4891b046312f4103344218f163ffd282e3915ecf09c6e1f5fceea6d6bad8f6 WatchSource:0}: Error finding container af4891b046312f4103344218f163ffd282e3915ecf09c6e1f5fceea6d6bad8f6: Status 404 returned error can't find the container with id af4891b046312f4103344218f163ffd282e3915ecf09c6e1f5fceea6d6bad8f6
Nov 03 00:44:15.314026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:15.313979    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn" event={"ID":"c746ad54-d687-422d-b0f4-3ba0e6f31a55","Type":"ContainerStarted","Data":"d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13"}
Nov 03 00:44:15.314487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:15.314047    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn" event={"ID":"c746ad54-d687-422d-b0f4-3ba0e6f31a55","Type":"ContainerStarted","Data":"af4891b046312f4103344218f163ffd282e3915ecf09c6e1f5fceea6d6bad8f6"}
Nov 03 00:44:15.314487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:15.314072    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:15.353746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:15.353695    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn" podStartSLOduration=2.353680802 podStartE2EDuration="2.353680802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:15.35035883 +0000 UTC m=+2292.905710854" watchObservedRunningTime="2026-11-03 00:44:15.353680802 +0000 UTC m=+2292.909032825"
Nov 03 00:44:17.197197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:17.197165    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:44:19.900699 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:44:19.900659    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:44:26.328426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:26.328394    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 00:44:40.717280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:40.713887    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:44:47.563636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.563603    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"]
Nov 03 00:44:47.568057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.568035    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.585656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.585634    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-2274efc91ed5\""
Nov 03 00:44:47.601564 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.601441    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"kube-root-ca.crt\""
Nov 03 00:44:47.601564 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.601446    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"routing-overlay\""
Nov 03 00:44:47.601564 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.601473    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdacdb\""
Nov 03 00:44:47.601564 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.601498    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-e2e--2e23f326-dockercfg-njdlx\""
Nov 03 00:44:47.601564 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.601533    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"openshift-service-ca.crt\""
Nov 03 00:44:47.609276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.605330    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-536b490d5ab1\""
Nov 03 00:44:47.609276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.607546    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"]
Nov 03 00:44:47.623732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.623699    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.623874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.623825    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c849m\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-kube-api-access-c849m\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.624121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.624090    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.624302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.624177    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.725379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.725346    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.725379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.725384    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c849m\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-kube-api-access-c849m\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.725584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.725468    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.725584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.725495    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.726113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.726092    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.726195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.726154    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.728308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.728249    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.744026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.743995    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c849m\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-kube-api-access-c849m\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:47.881801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:47.881769    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:48.187731 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:44:48.187694    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53ae069a_30d9_4b36_ac0a_1d4d21f272cd.slice/crio-4f71b57f38920aade6443d10635f23a4624a2190fcf27b4c978aae41f13b458f WatchSource:0}: Error finding container 4f71b57f38920aade6443d10635f23a4624a2190fcf27b4c978aae41f13b458f: Status 404 returned error can't find the container with id 4f71b57f38920aade6443d10635f23a4624a2190fcf27b4c978aae41f13b458f
Nov 03 00:44:48.188296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:48.188274    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"]
Nov 03 00:44:48.745706 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:48.745668    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l" event={"ID":"53ae069a-30d9-4b36-ac0a-1d4d21f272cd","Type":"ContainerStarted","Data":"72c441ae1e8c43d7bac021e54c98868caa36b83601b832ede5eb4b845253d5cf"}
Nov 03 00:44:48.747181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:48.745723    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l" event={"ID":"53ae069a-30d9-4b36-ac0a-1d4d21f272cd","Type":"ContainerStarted","Data":"4f71b57f38920aade6443d10635f23a4624a2190fcf27b4c978aae41f13b458f"}
Nov 03 00:44:48.747181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:48.745925    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:44:48.787692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:48.787636    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l" podStartSLOduration=1.787622928 podStartE2EDuration="1.787622928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:48.783919183 +0000 UTC m=+2326.339271208" watchObservedRunningTime="2026-11-03 00:44:48.787622928 +0000 UTC m=+2326.342974988"
Nov 03 00:44:49.372344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.372312    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"]
Nov 03 00:44:49.376300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.376241    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.394300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.393845    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-ac338d\""
Nov 03 00:44:49.395220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.395197    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"]
Nov 03 00:44:49.483319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.475531    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-ac338d-dockercfg-j74f2\""
Nov 03 00:44:49.512285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.475780    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7cxd\" (UniqueName: \"kubernetes.io/projected/c37bf3b6-0b50-4f34-837b-15368f5d6de7-kube-api-access-p7cxd\") pod \"payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.512285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.501656    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37bf3b6-0b50-4f34-837b-15368f5d6de7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.648807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.648717    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7cxd\" (UniqueName: \"kubernetes.io/projected/c37bf3b6-0b50-4f34-837b-15368f5d6de7-kube-api-access-p7cxd\") pod \"payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.648807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.648760    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37bf3b6-0b50-4f34-837b-15368f5d6de7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.649636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.649612    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37bf3b6-0b50-4f34-837b-15368f5d6de7-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.667581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.667552    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7cxd\" (UniqueName: \"kubernetes.io/projected/c37bf3b6-0b50-4f34-837b-15368f5d6de7-kube-api-access-p7cxd\") pod \"payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.691958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.691930    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:49.898872 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:44:49.898778    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:44:49.996065 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:44:49.996038    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc37bf3b6_0b50_4f34_837b_15368f5d6de7.slice/crio-8bf75618c816905595a1643561ebc7f5fc43612ed607a3976b50af77cf44c87d WatchSource:0}: Error finding container 8bf75618c816905595a1643561ebc7f5fc43612ed607a3976b50af77cf44c87d: Status 404 returned error can't find the container with id 8bf75618c816905595a1643561ebc7f5fc43612ed607a3976b50af77cf44c87d
Nov 03 00:44:49.996390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:49.996361    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"]
Nov 03 00:44:50.772190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:50.772152    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6" event={"ID":"c37bf3b6-0b50-4f34-837b-15368f5d6de7","Type":"ContainerStarted","Data":"883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991"}
Nov 03 00:44:50.772397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:50.772215    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6" event={"ID":"c37bf3b6-0b50-4f34-837b-15368f5d6de7","Type":"ContainerStarted","Data":"8bf75618c816905595a1643561ebc7f5fc43612ed607a3976b50af77cf44c87d"}
Nov 03 00:44:50.772397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:50.772288    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:44:50.809767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:50.809712    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6" podStartSLOduration=1.809697702 podStartE2EDuration="1.809697702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:44:50.807755006 +0000 UTC m=+2328.363107051" watchObservedRunningTime="2026-11-03 00:44:50.809697702 +0000 UTC m=+2328.365049725"
Nov 03 00:44:59.766461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:44:59.766426    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:45:00.252522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.252489    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"]
Nov 03 00:45:00.268290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.268225    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"]
Nov 03 00:45:00.268467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.268374    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.273613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.273575    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.298681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.298658    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"]
Nov 03 00:45:00.304841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.304812    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"]
Nov 03 00:45:00.308850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.308829    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 00:45:00.317153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.317112    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"]
Nov 03 00:45:00.323266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.323227    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.334178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.334108    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df147ebf-b21f-4df2-9e5d-05bcb412ca53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.338018 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.337990    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m7m6\" (UniqueName: \"kubernetes.io/projected/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-kube-api-access-4m7m6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.338224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.338203    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.338444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.338426    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd8xl\" (UniqueName: \"kubernetes.io/projected/df147ebf-b21f-4df2-9e5d-05bcb412ca53-kube-api-access-fd8xl\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.351608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.351555    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"]
Nov 03 00:45:00.366123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.366100    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"]
Nov 03 00:45:00.366648 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.366505    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.386051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.384436    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"]
Nov 03 00:45:00.386051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.384485    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"]
Nov 03 00:45:00.386051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.384592    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:00.389532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.389509    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"]
Nov 03 00:45:00.439939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.439906    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df147ebf-b21f-4df2-9e5d-05bcb412ca53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.440122 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.439954    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m7m6\" (UniqueName: \"kubernetes.io/projected/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-kube-api-access-4m7m6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.440122 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440009    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.440122 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440044    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777d72c8-be3c-49a9-8841-dc8aa04eecea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:00.440122 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440073    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d86kr\" (UniqueName: \"kubernetes.io/projected/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-kube-api-access-d86kr\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.440387 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440303    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd8xl\" (UniqueName: \"kubernetes.io/projected/df147ebf-b21f-4df2-9e5d-05bcb412ca53-kube-api-access-fd8xl\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.440387 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440346    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.440489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440449    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwlc9\" (UniqueName: \"kubernetes.io/projected/f1db35fa-8e4b-4908-9131-889f5efaf085-kube-api-access-kwlc9\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.440666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440643    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4qbs\" (UniqueName: \"kubernetes.io/projected/777d72c8-be3c-49a9-8841-dc8aa04eecea-kube-api-access-n4qbs\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:00.440742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440717    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1db35fa-8e4b-4908-9131-889f5efaf085-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.440797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440773    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df147ebf-b21f-4df2-9e5d-05bcb412ca53-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.440894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.440874    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.462374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.462348    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m7m6\" (UniqueName: \"kubernetes.io/projected/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-kube-api-access-4m7m6\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.462525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.462417    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd8xl\" (UniqueName: \"kubernetes.io/projected/df147ebf-b21f-4df2-9e5d-05bcb412ca53-kube-api-access-fd8xl\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.541974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.541895    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4qbs\" (UniqueName: \"kubernetes.io/projected/777d72c8-be3c-49a9-8841-dc8aa04eecea-kube-api-access-n4qbs\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:00.541974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.541946    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1db35fa-8e4b-4908-9131-889f5efaf085-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.542188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.542035    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777d72c8-be3c-49a9-8841-dc8aa04eecea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:00.542188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.542063    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d86kr\" (UniqueName: \"kubernetes.io/projected/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-kube-api-access-d86kr\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.542188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.542135    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.542374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.542207    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwlc9\" (UniqueName: \"kubernetes.io/projected/f1db35fa-8e4b-4908-9131-889f5efaf085-kube-api-access-kwlc9\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.542784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.542764    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1db35fa-8e4b-4908-9131-889f5efaf085-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.542936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.542913    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777d72c8-be3c-49a9-8841-dc8aa04eecea-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:00.542979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.542923    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.560815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.560790    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4qbs\" (UniqueName: \"kubernetes.io/projected/777d72c8-be3c-49a9-8841-dc8aa04eecea-kube-api-access-n4qbs\") pod \"maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:00.561681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.561658    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwlc9\" (UniqueName: \"kubernetes.io/projected/f1db35fa-8e4b-4908-9131-889f5efaf085-kube-api-access-kwlc9\") pod \"maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.561798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.561678    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d86kr\" (UniqueName: \"kubernetes.io/projected/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-kube-api-access-d86kr\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.601074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.601046    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:00.605989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.605970    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:00.644098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.644059    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:00.691105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.690770    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:00.704743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:00.704613    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:01.019661 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:01.019636    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1db35fa_8e4b_4908_9131_889f5efaf085.slice/crio-b1de8e7dc6323e0253628b4623c24ffaf5aa2157ca4460230bf62aa75979dd36 WatchSource:0}: Error finding container b1de8e7dc6323e0253628b4623c24ffaf5aa2157ca4460230bf62aa75979dd36: Status 404 returned error can't find the container with id b1de8e7dc6323e0253628b4623c24ffaf5aa2157ca4460230bf62aa75979dd36
Nov 03 00:45:01.021030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:01.021003    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"]
Nov 03 00:45:01.092202 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:01.092169    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ea3b26_88dc_48fb_844d_1d37365e9e3d.slice/crio-8f394932f4704b1dfba2c4a549220471afc0d6706486eeeef9172d387c2bbff8 WatchSource:0}: Error finding container 8f394932f4704b1dfba2c4a549220471afc0d6706486eeeef9172d387c2bbff8: Status 404 returned error can't find the container with id 8f394932f4704b1dfba2c4a549220471afc0d6706486eeeef9172d387c2bbff8
Nov 03 00:45:01.093646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:01.093622    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"]
Nov 03 00:45:01.093725 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:01.093642    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod777d72c8_be3c_49a9_8841_dc8aa04eecea.slice/crio-905e6674ea787009f88a154973239e9375336da7afca939f84ccd079f7197886 WatchSource:0}: Error finding container 905e6674ea787009f88a154973239e9375336da7afca939f84ccd079f7197886: Status 404 returned error can't find the container with id 905e6674ea787009f88a154973239e9375336da7afca939f84ccd079f7197886
Nov 03 00:45:01.097705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:01.097680    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"]
Nov 03 00:45:01.122954 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:01.122928    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"]
Nov 03 00:45:01.124150 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:01.124122    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf147ebf_b21f_4df2_9e5d_05bcb412ca53.slice/crio-779e9649c6d48e3005504c7ea7a7d10acbe74c54b70ac4233bb81591114c989d WatchSource:0}: Error finding container 779e9649c6d48e3005504c7ea7a7d10acbe74c54b70ac4233bb81591114c989d: Status 404 returned error can't find the container with id 779e9649c6d48e3005504c7ea7a7d10acbe74c54b70ac4233bb81591114c989d
Nov 03 00:45:01.125553 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:01.125520    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf000b77c_fc1d_4f65_b7be_b4a0b0814ac3.slice/crio-07417abb4e24ef969c2275a4c7febae1ae0d4377ad7fce77a71c647b2efe7748 WatchSource:0}: Error finding container 07417abb4e24ef969c2275a4c7febae1ae0d4377ad7fce77a71c647b2efe7748: Status 404 returned error can't find the container with id 07417abb4e24ef969c2275a4c7febae1ae0d4377ad7fce77a71c647b2efe7748
Nov 03 00:45:01.127393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:01.127368    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"]
Nov 03 00:45:01.783429 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:01.783400    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 00:45:02.050020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:02.049922    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" event={"ID":"c9ea3b26-88dc-48fb-844d-1d37365e9e3d","Type":"ContainerStarted","Data":"8f394932f4704b1dfba2c4a549220471afc0d6706486eeeef9172d387c2bbff8"}
Nov 03 00:45:02.052717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:02.052682    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerStarted","Data":"07417abb4e24ef969c2275a4c7febae1ae0d4377ad7fce77a71c647b2efe7748"}
Nov 03 00:45:02.054562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:02.054538    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerStarted","Data":"905e6674ea787009f88a154973239e9375336da7afca939f84ccd079f7197886"}
Nov 03 00:45:02.056435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:02.056410    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerStarted","Data":"b1de8e7dc6323e0253628b4623c24ffaf5aa2157ca4460230bf62aa75979dd36"}
Nov 03 00:45:02.058153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:02.058132    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerStarted","Data":"779e9649c6d48e3005504c7ea7a7d10acbe74c54b70ac4233bb81591114c989d"}
Nov 03 00:45:06.112060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.112025    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerStarted","Data":"8859a87c90f738ae6282155aa590c48f7467c0fb7f936056791c652b4f52a455"}
Nov 03 00:45:06.113800 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.113774    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerStarted","Data":"d0fd0dc4389761bcdbcb4cb022e239aef3b52a29cb4c517f2d0d735462f063a0"}
Nov 03 00:45:06.115557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.115532    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerStarted","Data":"512aba3a083aba4b48c0c18417af1e2b54f48d04a956a1689ba2f08d52eb25f1"}
Nov 03 00:45:06.117149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.117125    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerStarted","Data":"161e896c77c27efabced98b47179c17cfd74d8ae6cfe497795708cf11d08ec03"}
Nov 03 00:45:06.154748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.154643    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" podStartSLOduration=1.596354946 podStartE2EDuration="6.154629041s" totalImagesPullingTime="4.558274095s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:06.151882696 +0000 UTC m=+2343.707234723" watchObservedRunningTime="2026-11-03 00:45:06.154629041 +0000 UTC m=+2343.709981105"
Nov 03 00:45:06.191232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.191148    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" podStartSLOduration=1.894830405 podStartE2EDuration="6.191129128s" totalImagesPullingTime="4.296298723s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:06.18765908 +0000 UTC m=+2343.743011105" watchObservedRunningTime="2026-11-03 00:45:06.191129128 +0000 UTC m=+2343.746481154"
Nov 03 00:45:06.226046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.225995    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" podStartSLOduration=1.564113213 podStartE2EDuration="6.225981129s" totalImagesPullingTime="4.661867916s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:06.223229941 +0000 UTC m=+2343.778581965" watchObservedRunningTime="2026-11-03 00:45:06.225981129 +0000 UTC m=+2343.781333153"
Nov 03 00:45:06.260711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:06.260655    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" podStartSLOduration=1.957278616 podStartE2EDuration="6.260640264s" totalImagesPullingTime="4.303361648s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:06.257823287 +0000 UTC m=+2343.813175313" watchObservedRunningTime="2026-11-03 00:45:06.260640264 +0000 UTC m=+2343.815992287"
Nov 03 00:45:07.125702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.125667    6000 generic.go:356] "Generic (PLEG): container finished" podID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerID="2ddeb58d98d1bba77715fe04e7602465a8aea6927b4351b08c7db56594e44276" exitCode=60
Nov 03 00:45:07.126204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.125764    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" event={"ID":"c9ea3b26-88dc-48fb-844d-1d37365e9e3d","Type":"ContainerDied","Data":"2ddeb58d98d1bba77715fe04e7602465a8aea6927b4351b08c7db56594e44276"}
Nov 03 00:45:07.126204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.126056    6000 scope.go:122] "RemoveContainer" containerID="2ddeb58d98d1bba77715fe04e7602465a8aea6927b4351b08c7db56594e44276"
Nov 03 00:45:07.127670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.127650    6000 generic.go:356] "Generic (PLEG): container finished" podID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerID="d0fd0dc4389761bcdbcb4cb022e239aef3b52a29cb4c517f2d0d735462f063a0" exitCode=60
Nov 03 00:45:07.127887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.127857    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerDied","Data":"d0fd0dc4389761bcdbcb4cb022e239aef3b52a29cb4c517f2d0d735462f063a0"}
Nov 03 00:45:07.128196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.128181    6000 scope.go:122] "RemoveContainer" containerID="d0fd0dc4389761bcdbcb4cb022e239aef3b52a29cb4c517f2d0d735462f063a0"
Nov 03 00:45:07.129671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.129647    6000 generic.go:356] "Generic (PLEG): container finished" podID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerID="512aba3a083aba4b48c0c18417af1e2b54f48d04a956a1689ba2f08d52eb25f1" exitCode=60
Nov 03 00:45:07.129778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.129753    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerDied","Data":"512aba3a083aba4b48c0c18417af1e2b54f48d04a956a1689ba2f08d52eb25f1"}
Nov 03 00:45:07.130076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.130057    6000 scope.go:122] "RemoveContainer" containerID="512aba3a083aba4b48c0c18417af1e2b54f48d04a956a1689ba2f08d52eb25f1"
Nov 03 00:45:07.132334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.132300    6000 generic.go:356] "Generic (PLEG): container finished" podID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerID="161e896c77c27efabced98b47179c17cfd74d8ae6cfe497795708cf11d08ec03" exitCode=60
Nov 03 00:45:07.132439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.132403    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerDied","Data":"161e896c77c27efabced98b47179c17cfd74d8ae6cfe497795708cf11d08ec03"}
Nov 03 00:45:07.132585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.132569    6000 scope.go:122] "RemoveContainer" containerID="161e896c77c27efabced98b47179c17cfd74d8ae6cfe497795708cf11d08ec03"
Nov 03 00:45:07.134210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.134190    6000 generic.go:356] "Generic (PLEG): container finished" podID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerID="8859a87c90f738ae6282155aa590c48f7467c0fb7f936056791c652b4f52a455" exitCode=60
Nov 03 00:45:07.134336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.134243    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerDied","Data":"8859a87c90f738ae6282155aa590c48f7467c0fb7f936056791c652b4f52a455"}
Nov 03 00:45:07.134655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:07.134499    6000 scope.go:122] "RemoveContainer" containerID="8859a87c90f738ae6282155aa590c48f7467c0fb7f936056791c652b4f52a455"
Nov 03 00:45:08.145033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:08.144991    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" event={"ID":"c9ea3b26-88dc-48fb-844d-1d37365e9e3d","Type":"ContainerStarted","Data":"629260ac8a56645f4107e638331b6955025c793c64c8d015d1934abcc7850988"}
Nov 03 00:45:08.147795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:08.147757    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerStarted","Data":"772ab2fe181d89e99856cb23e01e9c45625f24108e67f03ff6a9594a36adb313"}
Nov 03 00:45:08.150763 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:08.150425    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerStarted","Data":"46eab1aa5931a9baf83c1e6ee1cb2fc61687ff66e62818b465bf602e84b41af0"}
Nov 03 00:45:08.153047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:08.153020    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerStarted","Data":"635c9626221c29d1e69ecb25f227aa191012b496dc3a3940d61161af093aad88"}
Nov 03 00:45:08.155042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:08.155022    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerStarted","Data":"a4e954acdc1103dae9c836d03b61fc3e59360ba64f680eaac0d7c5917d9f77fe"}
Nov 03 00:45:08.189559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:08.189500    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" podStartSLOduration=3.633625479 podStartE2EDuration="8.189487246s" totalImagesPullingTime="4.555861767s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:08.185393139 +0000 UTC m=+2345.740745163" watchObservedRunningTime="2026-11-03 00:45:08.189487246 +0000 UTC m=+2345.744839270"
Nov 03 00:45:09.164062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.164031    6000 generic.go:356] "Generic (PLEG): container finished" podID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerID="635c9626221c29d1e69ecb25f227aa191012b496dc3a3940d61161af093aad88" exitCode=60
Nov 03 00:45:09.164571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.164112    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerDied","Data":"635c9626221c29d1e69ecb25f227aa191012b496dc3a3940d61161af093aad88"}
Nov 03 00:45:09.164571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.164171    6000 scope.go:122] "RemoveContainer" containerID="161e896c77c27efabced98b47179c17cfd74d8ae6cfe497795708cf11d08ec03"
Nov 03 00:45:09.164571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.164460    6000 scope.go:122] "RemoveContainer" containerID="635c9626221c29d1e69ecb25f227aa191012b496dc3a3940d61161af093aad88"
Nov 03 00:45:09.164991 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:09.164966    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk_odh-ai-gateway-infra(f1db35fa-8e4b-4908-9131-889f5efaf085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085"
Nov 03 00:45:09.166414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.166394    6000 generic.go:356] "Generic (PLEG): container finished" podID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerID="a4e954acdc1103dae9c836d03b61fc3e59360ba64f680eaac0d7c5917d9f77fe" exitCode=60
Nov 03 00:45:09.166499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.166458    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerDied","Data":"a4e954acdc1103dae9c836d03b61fc3e59360ba64f680eaac0d7c5917d9f77fe"}
Nov 03 00:45:09.166771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.166755    6000 scope.go:122] "RemoveContainer" containerID="a4e954acdc1103dae9c836d03b61fc3e59360ba64f680eaac0d7c5917d9f77fe"
Nov 03 00:45:09.167177 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:09.167150    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx_odh-ai-gateway-infra(df147ebf-b21f-4df2-9e5d-05bcb412ca53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53"
Nov 03 00:45:09.168615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.168593    6000 generic.go:356] "Generic (PLEG): container finished" podID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerID="629260ac8a56645f4107e638331b6955025c793c64c8d015d1934abcc7850988" exitCode=60
Nov 03 00:45:09.168707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.168668    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" event={"ID":"c9ea3b26-88dc-48fb-844d-1d37365e9e3d","Type":"ContainerDied","Data":"629260ac8a56645f4107e638331b6955025c793c64c8d015d1934abcc7850988"}
Nov 03 00:45:09.168891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.168876    6000 scope.go:122] "RemoveContainer" containerID="629260ac8a56645f4107e638331b6955025c793c64c8d015d1934abcc7850988"
Nov 03 00:45:09.169144 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:09.169122    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf_odh-ai-gateway-infra(c9ea3b26-88dc-48fb-844d-1d37365e9e3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d"
Nov 03 00:45:09.170590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.170568    6000 generic.go:356] "Generic (PLEG): container finished" podID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerID="772ab2fe181d89e99856cb23e01e9c45625f24108e67f03ff6a9594a36adb313" exitCode=60
Nov 03 00:45:09.170697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.170657    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerDied","Data":"772ab2fe181d89e99856cb23e01e9c45625f24108e67f03ff6a9594a36adb313"}
Nov 03 00:45:09.170889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.170875    6000 scope.go:122] "RemoveContainer" containerID="772ab2fe181d89e99856cb23e01e9c45625f24108e67f03ff6a9594a36adb313"
Nov 03 00:45:09.171139 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:09.171119    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv_odh-ai-gateway-infra(f000b77c-fc1d-4f65-b7be-b4a0b0814ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3"
Nov 03 00:45:09.172759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.172735    6000 generic.go:356] "Generic (PLEG): container finished" podID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerID="46eab1aa5931a9baf83c1e6ee1cb2fc61687ff66e62818b465bf602e84b41af0" exitCode=60
Nov 03 00:45:09.172863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.172810    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerDied","Data":"46eab1aa5931a9baf83c1e6ee1cb2fc61687ff66e62818b465bf602e84b41af0"}
Nov 03 00:45:09.173041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.173027    6000 scope.go:122] "RemoveContainer" containerID="46eab1aa5931a9baf83c1e6ee1cb2fc61687ff66e62818b465bf602e84b41af0"
Nov 03 00:45:09.173346 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:09.173326    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr_odh-ai-gateway-infra(777d72c8-be3c-49a9-8841-dc8aa04eecea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea"
Nov 03 00:45:09.178454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.178436    6000 scope.go:122] "RemoveContainer" containerID="8859a87c90f738ae6282155aa590c48f7467c0fb7f936056791c652b4f52a455"
Nov 03 00:45:09.192544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.192525    6000 scope.go:122] "RemoveContainer" containerID="2ddeb58d98d1bba77715fe04e7602465a8aea6927b4351b08c7db56594e44276"
Nov 03 00:45:09.206857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.206841    6000 scope.go:122] "RemoveContainer" containerID="d0fd0dc4389761bcdbcb4cb022e239aef3b52a29cb4c517f2d0d735462f063a0"
Nov 03 00:45:09.220439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:09.220419    6000 scope.go:122] "RemoveContainer" containerID="512aba3a083aba4b48c0c18417af1e2b54f48d04a956a1689ba2f08d52eb25f1"
Nov 03 00:45:10.183395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:10.183358    6000 scope.go:122] "RemoveContainer" containerID="629260ac8a56645f4107e638331b6955025c793c64c8d015d1934abcc7850988"
Nov 03 00:45:10.183909 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:10.183698    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf_odh-ai-gateway-infra(c9ea3b26-88dc-48fb-844d-1d37365e9e3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d"
Nov 03 00:45:10.185313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:10.185289    6000 scope.go:122] "RemoveContainer" containerID="772ab2fe181d89e99856cb23e01e9c45625f24108e67f03ff6a9594a36adb313"
Nov 03 00:45:10.185571 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:10.185551    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv_odh-ai-gateway-infra(f000b77c-fc1d-4f65-b7be-b4a0b0814ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3"
Nov 03 00:45:10.186964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:10.186945    6000 scope.go:122] "RemoveContainer" containerID="46eab1aa5931a9baf83c1e6ee1cb2fc61687ff66e62818b465bf602e84b41af0"
Nov 03 00:45:10.187226 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:10.187207    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr_odh-ai-gateway-infra(777d72c8-be3c-49a9-8841-dc8aa04eecea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea"
Nov 03 00:45:10.188633 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:10.188615    6000 scope.go:122] "RemoveContainer" containerID="635c9626221c29d1e69ecb25f227aa191012b496dc3a3940d61161af093aad88"
Nov 03 00:45:10.188838 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:10.188820    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk_odh-ai-gateway-infra(f1db35fa-8e4b-4908-9131-889f5efaf085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085"
Nov 03 00:45:10.190666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:10.190649    6000 scope.go:122] "RemoveContainer" containerID="a4e954acdc1103dae9c836d03b61fc3e59360ba64f680eaac0d7c5917d9f77fe"
Nov 03 00:45:10.190851 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:10.190835    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx_odh-ai-gateway-infra(df147ebf-b21f-4df2-9e5d-05bcb412ca53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53"
Nov 03 00:45:14.055551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.055468    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"]
Nov 03 00:45:14.062109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.062085    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.080095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.080071    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"]
Nov 03 00:45:14.087011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.086983    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"agc-cred-2e7d174e9f98\""
Nov 03 00:45:14.087142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.087060    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"payload-processing-external-model-e2e--4be0dbee-dockercfg-tqf75\""
Nov 03 00:45:14.087195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.087145    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"payload-processing-external-model-plugins-e2e-extmod-b-909268\""
Nov 03 00:45:14.087195 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.087146    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"openshift-service-ca.crt\""
Nov 03 00:45:14.087935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.087912    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"kube-root-ca.crt\""
Nov 03 00:45:14.087935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.087922    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"routing-overlay\""
Nov 03 00:45:14.220287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.218714    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.220287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.218850    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.220287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.218937    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.220287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.219051    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jkd4\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-kube-api-access-8jkd4\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.322153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.322054    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.322497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.322475    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.322661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.322646    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.322860 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.322846    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jkd4\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-kube-api-access-8jkd4\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.324022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.323993    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.324303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.324234    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.330707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.330673    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.343058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.343017    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jkd4\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-kube-api-access-8jkd4\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548drpvnf\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.377593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.377565    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:14.687210 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:14.687180    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98f73ad_617e_4322_b81f_60bef48a13be.slice/crio-46c031685a458691534e19ca9b505bb6b54bc16196071a5a64b202e2c3b97080 WatchSource:0}: Error finding container 46c031685a458691534e19ca9b505bb6b54bc16196071a5a64b202e2c3b97080: Status 404 returned error can't find the container with id 46c031685a458691534e19ca9b505bb6b54bc16196071a5a64b202e2c3b97080
Nov 03 00:45:14.688193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:14.688173    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"]
Nov 03 00:45:15.342784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:15.342751    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf" event={"ID":"f98f73ad-617e-4322-b81f-60bef48a13be","Type":"ContainerStarted","Data":"88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76"}
Nov 03 00:45:15.343305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:15.342802    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf" event={"ID":"f98f73ad-617e-4322-b81f-60bef48a13be","Type":"ContainerStarted","Data":"46c031685a458691534e19ca9b505bb6b54bc16196071a5a64b202e2c3b97080"}
Nov 03 00:45:15.343305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:15.342891    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:15.385734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:15.385684    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf" podStartSLOduration=1.38566919 podStartE2EDuration="1.38566919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:15.381334014 +0000 UTC m=+2352.936686051" watchObservedRunningTime="2026-11-03 00:45:15.38566919 +0000 UTC m=+2352.941021214"
Nov 03 00:45:19.560068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:19.560039    6000 scope.go:122] "RemoveContainer" containerID="46eab1aa5931a9baf83c1e6ee1cb2fc61687ff66e62818b465bf602e84b41af0"
Nov 03 00:45:19.898470 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:19.898435    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:45:20.383068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:20.383032    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerStarted","Data":"b0e987e258a8ae65366c19964e2195b0099a922473b324f60dd6ac17ecba6425"}
Nov 03 00:45:21.395603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.395563    6000 generic.go:356] "Generic (PLEG): container finished" podID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerID="b0e987e258a8ae65366c19964e2195b0099a922473b324f60dd6ac17ecba6425" exitCode=60
Nov 03 00:45:21.396193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.395657    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerDied","Data":"b0e987e258a8ae65366c19964e2195b0099a922473b324f60dd6ac17ecba6425"}
Nov 03 00:45:21.396193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.395743    6000 scope.go:122] "RemoveContainer" containerID="46eab1aa5931a9baf83c1e6ee1cb2fc61687ff66e62818b465bf602e84b41af0"
Nov 03 00:45:21.396193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.396178    6000 scope.go:122] "RemoveContainer" containerID="b0e987e258a8ae65366c19964e2195b0099a922473b324f60dd6ac17ecba6425"
Nov 03 00:45:21.396646 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:21.396624    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr_odh-ai-gateway-infra(777d72c8-be3c-49a9-8841-dc8aa04eecea)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea"
Nov 03 00:45:21.432301 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.432249    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"]
Nov 03 00:45:21.559863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.559830    6000 scope.go:122] "RemoveContainer" containerID="a4e954acdc1103dae9c836d03b61fc3e59360ba64f680eaac0d7c5917d9f77fe"
Nov 03 00:45:21.560366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.560343    6000 scope.go:122] "RemoveContainer" containerID="635c9626221c29d1e69ecb25f227aa191012b496dc3a3940d61161af093aad88"
Nov 03 00:45:21.560489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:21.560428    6000 scope.go:122] "RemoveContainer" containerID="772ab2fe181d89e99856cb23e01e9c45625f24108e67f03ff6a9594a36adb313"
Nov 03 00:45:22.427742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.427706    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerStarted","Data":"aed2a393a9478acbcc42b1af9e9a51df5842437b1abefc153d7a1fb5312fbfd6"}
Nov 03 00:45:22.445672 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.445629    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerStarted","Data":"831e59f7211e71708b898eef3e22a4894ea5a1c8112d221e1e87d4ac51e90790"}
Nov 03 00:45:22.447767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.447741    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerStarted","Data":"0f0fd6d90bf14e00463b2f203f0d97c9374d50de240510b0a20eaf551d2ce12d"}
Nov 03 00:45:22.561304 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.561244    6000 scope.go:122] "RemoveContainer" containerID="629260ac8a56645f4107e638331b6955025c793c64c8d015d1934abcc7850988"
Nov 03 00:45:22.831118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.831093    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:22.906740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.906708    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/777d72c8-be3c-49a9-8841-dc8aa04eecea-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777d72c8-be3c-49a9-8841-dc8aa04eecea-maas-api-service-ca\") pod \"777d72c8-be3c-49a9-8841-dc8aa04eecea\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") "
Nov 03 00:45:22.906890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.906853    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/777d72c8-be3c-49a9-8841-dc8aa04eecea-kube-api-access-n4qbs\" (UniqueName: \"kubernetes.io/projected/777d72c8-be3c-49a9-8841-dc8aa04eecea-kube-api-access-n4qbs\") pod \"777d72c8-be3c-49a9-8841-dc8aa04eecea\" (UID: \"777d72c8-be3c-49a9-8841-dc8aa04eecea\") "
Nov 03 00:45:22.907194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.907161    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/777d72c8-be3c-49a9-8841-dc8aa04eecea-maas-api-service-ca" pod "777d72c8-be3c-49a9-8841-dc8aa04eecea" (UID: "777d72c8-be3c-49a9-8841-dc8aa04eecea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:22.907535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.907511    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/777d72c8-be3c-49a9-8841-dc8aa04eecea-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:22.988112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:22.988032    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/777d72c8-be3c-49a9-8841-dc8aa04eecea-kube-api-access-n4qbs" pod "777d72c8-be3c-49a9-8841-dc8aa04eecea" (UID: "777d72c8-be3c-49a9-8841-dc8aa04eecea"). InnerVolumeSpecName "kube-api-access-n4qbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:23.009371 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.009337    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4qbs\" (UniqueName: \"kubernetes.io/projected/777d72c8-be3c-49a9-8841-dc8aa04eecea-kube-api-access-n4qbs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:23.457756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.457725    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" event={"ID":"c9ea3b26-88dc-48fb-844d-1d37365e9e3d","Type":"ContainerStarted","Data":"860c9390a15012da0a7f4ef25eaa7b4d4035bcdc6c96e2a5281ac7d21a076636"}
Nov 03 00:45:23.459759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.459736    6000 generic.go:356] "Generic (PLEG): container finished" podID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerID="aed2a393a9478acbcc42b1af9e9a51df5842437b1abefc153d7a1fb5312fbfd6" exitCode=60
Nov 03 00:45:23.459903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.459801    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerDied","Data":"aed2a393a9478acbcc42b1af9e9a51df5842437b1abefc153d7a1fb5312fbfd6"}
Nov 03 00:45:23.459903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.459857    6000 scope.go:122] "RemoveContainer" containerID="772ab2fe181d89e99856cb23e01e9c45625f24108e67f03ff6a9594a36adb313"
Nov 03 00:45:23.460327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.460304    6000 scope.go:122] "RemoveContainer" containerID="aed2a393a9478acbcc42b1af9e9a51df5842437b1abefc153d7a1fb5312fbfd6"
Nov 03 00:45:23.460693 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:23.460671    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv_odh-ai-gateway-infra(f000b77c-fc1d-4f65-b7be-b4a0b0814ac3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3"
Nov 03 00:45:23.461305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.461283    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr" event={"ID":"777d72c8-be3c-49a9-8841-dc8aa04eecea","Type":"ContainerDied","Data":"905e6674ea787009f88a154973239e9375336da7afca939f84ccd079f7197886"}
Nov 03 00:45:23.461394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.461306    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"
Nov 03 00:45:23.463438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.463418    6000 generic.go:356] "Generic (PLEG): container finished" podID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerID="831e59f7211e71708b898eef3e22a4894ea5a1c8112d221e1e87d4ac51e90790" exitCode=60
Nov 03 00:45:23.463549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.463500    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerDied","Data":"831e59f7211e71708b898eef3e22a4894ea5a1c8112d221e1e87d4ac51e90790"}
Nov 03 00:45:23.463865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.463847    6000 scope.go:122] "RemoveContainer" containerID="831e59f7211e71708b898eef3e22a4894ea5a1c8112d221e1e87d4ac51e90790"
Nov 03 00:45:23.464220 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:23.464201    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk_odh-ai-gateway-infra(f1db35fa-8e4b-4908-9131-889f5efaf085)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085"
Nov 03 00:45:23.465659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.465636    6000 generic.go:356] "Generic (PLEG): container finished" podID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerID="0f0fd6d90bf14e00463b2f203f0d97c9374d50de240510b0a20eaf551d2ce12d" exitCode=60
Nov 03 00:45:23.465746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.465683    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerDied","Data":"0f0fd6d90bf14e00463b2f203f0d97c9374d50de240510b0a20eaf551d2ce12d"}
Nov 03 00:45:23.466070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.466054    6000 scope.go:122] "RemoveContainer" containerID="0f0fd6d90bf14e00463b2f203f0d97c9374d50de240510b0a20eaf551d2ce12d"
Nov 03 00:45:23.466402 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:23.466381    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx_odh-ai-gateway-infra(df147ebf-b21f-4df2-9e5d-05bcb412ca53)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53"
Nov 03 00:45:23.475927 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.475908    6000 scope.go:122] "RemoveContainer" containerID="b0e987e258a8ae65366c19964e2195b0099a922473b324f60dd6ac17ecba6425"
Nov 03 00:45:23.486038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.486005    6000 scope.go:122] "RemoveContainer" containerID="635c9626221c29d1e69ecb25f227aa191012b496dc3a3940d61161af093aad88"
Nov 03 00:45:23.488322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.488300    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"]
Nov 03 00:45:23.500835 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.500811    6000 scope.go:122] "RemoveContainer" containerID="a4e954acdc1103dae9c836d03b61fc3e59360ba64f680eaac0d7c5917d9f77fe"
Nov 03 00:45:23.520952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.520928    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"]
Nov 03 00:45:23.559891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.559858    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"]
Nov 03 00:45:23.659926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.659890    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"]
Nov 03 00:45:23.664325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:23.664297    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-fdacdb-29894445-4gznr"]
Nov 03 00:45:24.474295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.474199    6000 generic.go:356] "Generic (PLEG): container finished" podID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerID="860c9390a15012da0a7f4ef25eaa7b4d4035bcdc6c96e2a5281ac7d21a076636" exitCode=60
Nov 03 00:45:24.474295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.474284    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" event={"ID":"c9ea3b26-88dc-48fb-844d-1d37365e9e3d","Type":"ContainerDied","Data":"860c9390a15012da0a7f4ef25eaa7b4d4035bcdc6c96e2a5281ac7d21a076636"}
Nov 03 00:45:24.474804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.474347    6000 scope.go:122] "RemoveContainer" containerID="629260ac8a56645f4107e638331b6955025c793c64c8d015d1934abcc7850988"
Nov 03 00:45:24.474804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.474764    6000 scope.go:122] "RemoveContainer" containerID="860c9390a15012da0a7f4ef25eaa7b4d4035bcdc6c96e2a5281ac7d21a076636"
Nov 03 00:45:24.475169 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:24.475144    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf_odh-ai-gateway-infra(c9ea3b26-88dc-48fb-844d-1d37365e9e3d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d"
Nov 03 00:45:24.529707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.529675    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"]
Nov 03 00:45:24.583327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.583295    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" path="/var/lib/kubelet/pods/777d72c8-be3c-49a9-8841-dc8aa04eecea/volumes"
Nov 03 00:45:24.696069 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.696040    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:24.754024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.753999    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:24.761817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.761794    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:24.841531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.841495    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-maas-api-service-ca\") pod \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") "
Nov 03 00:45:24.841762 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.841676    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-kube-api-access-4m7m6\" (UniqueName: \"kubernetes.io/projected/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-kube-api-access-4m7m6\") pod \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\" (UID: \"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3\") "
Nov 03 00:45:24.841905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.841882    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df147ebf-b21f-4df2-9e5d-05bcb412ca53-kube-api-access-fd8xl\" (UniqueName: \"kubernetes.io/projected/df147ebf-b21f-4df2-9e5d-05bcb412ca53-kube-api-access-fd8xl\") pod \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") "
Nov 03 00:45:24.845242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.842016    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df147ebf-b21f-4df2-9e5d-05bcb412ca53-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df147ebf-b21f-4df2-9e5d-05bcb412ca53-maas-api-service-ca\") pod \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\" (UID: \"df147ebf-b21f-4df2-9e5d-05bcb412ca53\") "
Nov 03 00:45:24.845242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.842964    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df147ebf-b21f-4df2-9e5d-05bcb412ca53-maas-api-service-ca" pod "df147ebf-b21f-4df2-9e5d-05bcb412ca53" (UID: "df147ebf-b21f-4df2-9e5d-05bcb412ca53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:24.845242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.843645    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-maas-api-service-ca" pod "f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" (UID: "f000b77c-fc1d-4f65-b7be-b4a0b0814ac3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:24.845977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.845944    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-kube-api-access-4m7m6" pod "f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" (UID: "f000b77c-fc1d-4f65-b7be-b4a0b0814ac3"). InnerVolumeSpecName "kube-api-access-4m7m6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:24.846552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.846527    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df147ebf-b21f-4df2-9e5d-05bcb412ca53-kube-api-access-fd8xl" pod "df147ebf-b21f-4df2-9e5d-05bcb412ca53" (UID: "df147ebf-b21f-4df2-9e5d-05bcb412ca53"). InnerVolumeSpecName "kube-api-access-fd8xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:24.943216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.943182    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1db35fa-8e4b-4908-9131-889f5efaf085-kube-api-access-kwlc9\" (UniqueName: \"kubernetes.io/projected/f1db35fa-8e4b-4908-9131-889f5efaf085-kube-api-access-kwlc9\") pod \"f1db35fa-8e4b-4908-9131-889f5efaf085\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") "
Nov 03 00:45:24.943473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.943453    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1db35fa-8e4b-4908-9131-889f5efaf085-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1db35fa-8e4b-4908-9131-889f5efaf085-maas-api-service-ca\") pod \"f1db35fa-8e4b-4908-9131-889f5efaf085\" (UID: \"f1db35fa-8e4b-4908-9131-889f5efaf085\") "
Nov 03 00:45:24.943879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.943853    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1db35fa-8e4b-4908-9131-889f5efaf085-maas-api-service-ca" pod "f1db35fa-8e4b-4908-9131-889f5efaf085" (UID: "f1db35fa-8e4b-4908-9131-889f5efaf085"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:24.944059 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.944043    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd8xl\" (UniqueName: \"kubernetes.io/projected/df147ebf-b21f-4df2-9e5d-05bcb412ca53-kube-api-access-fd8xl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:24.944096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.944062    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1db35fa-8e4b-4908-9131-889f5efaf085-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:24.944096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.944072    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df147ebf-b21f-4df2-9e5d-05bcb412ca53-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:24.944096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.944080    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:24.944096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.944089    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m7m6\" (UniqueName: \"kubernetes.io/projected/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3-kube-api-access-4m7m6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:24.945629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:24.945605    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1db35fa-8e4b-4908-9131-889f5efaf085-kube-api-access-kwlc9" pod "f1db35fa-8e4b-4908-9131-889f5efaf085" (UID: "f1db35fa-8e4b-4908-9131-889f5efaf085"). InnerVolumeSpecName "kube-api-access-kwlc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:25.045834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.045757    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwlc9\" (UniqueName: \"kubernetes.io/projected/f1db35fa-8e4b-4908-9131-889f5efaf085-kube-api-access-kwlc9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:25.493897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.493870    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"
Nov 03 00:45:25.494327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.494298    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv" event={"ID":"f000b77c-fc1d-4f65-b7be-b4a0b0814ac3","Type":"ContainerDied","Data":"07417abb4e24ef969c2275a4c7febae1ae0d4377ad7fce77a71c647b2efe7748"}
Nov 03 00:45:25.494466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.494446    6000 scope.go:122] "RemoveContainer" containerID="aed2a393a9478acbcc42b1af9e9a51df5842437b1abefc153d7a1fb5312fbfd6"
Nov 03 00:45:25.496112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.496087    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"
Nov 03 00:45:25.496249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.496218    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk" event={"ID":"f1db35fa-8e4b-4908-9131-889f5efaf085","Type":"ContainerDied","Data":"b1de8e7dc6323e0253628b4623c24ffaf5aa2157ca4460230bf62aa75979dd36"}
Nov 03 00:45:25.497977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.497947    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx" event={"ID":"df147ebf-b21f-4df2-9e5d-05bcb412ca53","Type":"ContainerDied","Data":"779e9649c6d48e3005504c7ea7a7d10acbe74c54b70ac4233bb81591114c989d"}
Nov 03 00:45:25.498079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.498034    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"
Nov 03 00:45:25.509249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.509228    6000 scope.go:122] "RemoveContainer" containerID="831e59f7211e71708b898eef3e22a4894ea5a1c8112d221e1e87d4ac51e90790"
Nov 03 00:45:25.525429 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.525406    6000 scope.go:122] "RemoveContainer" containerID="0f0fd6d90bf14e00463b2f203f0d97c9374d50de240510b0a20eaf551d2ce12d"
Nov 03 00:45:25.579514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.570319    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"]
Nov 03 00:45:25.591743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.591710    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d9e067-29894445-kv6sk"]
Nov 03 00:45:25.623312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.622634    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"]
Nov 03 00:45:25.627862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.627833    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-099583-29894445-zcvxv"]
Nov 03 00:45:25.660968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.660936    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"]
Nov 03 00:45:25.683819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.683790    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894445-jl2sx"]
Nov 03 00:45:25.705200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.705167    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:25.858005 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.857971    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-kube-api-access-d86kr\" (UniqueName: \"kubernetes.io/projected/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-kube-api-access-d86kr\") pod \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") "
Nov 03 00:45:25.858192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.858161    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-maas-api-service-ca\") pod \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\" (UID: \"c9ea3b26-88dc-48fb-844d-1d37365e9e3d\") "
Nov 03 00:45:25.858549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.858520    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-maas-api-service-ca" pod "c9ea3b26-88dc-48fb-844d-1d37365e9e3d" (UID: "c9ea3b26-88dc-48fb-844d-1d37365e9e3d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:25.858790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.858768    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:25.860420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.860395    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-kube-api-access-d86kr" pod "c9ea3b26-88dc-48fb-844d-1d37365e9e3d" (UID: "c9ea3b26-88dc-48fb-844d-1d37365e9e3d"). InnerVolumeSpecName "kube-api-access-d86kr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:25.960155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:25.960118    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d86kr\" (UniqueName: \"kubernetes.io/projected/c9ea3b26-88dc-48fb-844d-1d37365e9e3d-kube-api-access-d86kr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:26.353726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.353697    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:45:26.514643 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.514612    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"
Nov 03 00:45:26.515118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.514610    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf" event={"ID":"c9ea3b26-88dc-48fb-844d-1d37365e9e3d","Type":"ContainerDied","Data":"8f394932f4704b1dfba2c4a549220471afc0d6706486eeeef9172d387c2bbff8"}
Nov 03 00:45:26.515118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.514744    6000 scope.go:122] "RemoveContainer" containerID="860c9390a15012da0a7f4ef25eaa7b4d4035bcdc6c96e2a5281ac7d21a076636"
Nov 03 00:45:26.570850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.570822    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" path="/var/lib/kubelet/pods/df147ebf-b21f-4df2-9e5d-05bcb412ca53/volumes"
Nov 03 00:45:26.571297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.571282    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" path="/var/lib/kubelet/pods/f000b77c-fc1d-4f65-b7be-b4a0b0814ac3/volumes"
Nov 03 00:45:26.571728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.571714    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" path="/var/lib/kubelet/pods/f1db35fa-8e4b-4908-9131-889f5efaf085/volumes"
Nov 03 00:45:26.572392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.572375    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"]
Nov 03 00:45:26.572474 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:26.572406    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894445-cxtzf"]
Nov 03 00:45:28.569130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:28.569098    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" path="/var/lib/kubelet/pods/c9ea3b26-88dc-48fb-844d-1d37365e9e3d/volumes"
Nov 03 00:45:31.718853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:31.718775    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:45:37.578536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.578505    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"]
Nov 03 00:45:37.579353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579337    6000 cpu_manager.go:414] "containerMap: removing container" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579355    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579369    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579374    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579383    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579388    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579398    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579403    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579409    6000 cpu_manager.go:414] "containerMap: removing container" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:37.579414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579414    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579427    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579431    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579458    6000 cpu_manager.go:414] "containerMap: removing container" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579463    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579469    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579473    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579481    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579486    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579497    6000 cpu_manager.go:414] "containerMap: removing container" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579502    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579508    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579512    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579629    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579637    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579642    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579648    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579654    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579659    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579663    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579668    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579674    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579680    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579686    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579692    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:37.579691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.579699    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:37.585322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.585299    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.607456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.607430    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"]
Nov 03 00:45:37.614295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.614250    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"agc-cred-c92123aa0995\""
Nov 03 00:45:37.687995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.687952    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.688187 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.688067    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.688243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.688186    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7l22\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-kube-api-access-v7l22\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.688243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.688233    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.785079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.785040    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"]
Nov 03 00:45:37.785330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.785307    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" containerID="cri-o://8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91" gracePeriod=30
Nov 03 00:45:37.789654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.789630    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.789780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.789710    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.789780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.789747    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7l22\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-kube-api-access-v7l22\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.789780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.789770    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.790483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.790453    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.790644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.790621    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.792528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.792509    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.814343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.814316    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7l22\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-kube-api-access-v7l22\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:37.883088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.883054    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"]
Nov 03 00:45:37.884033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.883288    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" containerID="cri-o://b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790" gracePeriod=30
Nov 03 00:45:37.900603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:37.900574    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:38.221285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:38.220404    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"]
Nov 03 00:45:38.221285 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:38.220867    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6ad117_5d8f_4554_bbcb_a77dd74fc9ae.slice/crio-a33aa79688a40de926a63f738569542dee1c1be6f5f287fe1de7db9c3cf23bbe WatchSource:0}: Error finding container a33aa79688a40de926a63f738569542dee1c1be6f5f287fe1de7db9c3cf23bbe: Status 404 returned error can't find the container with id a33aa79688a40de926a63f738569542dee1c1be6f5f287fe1de7db9c3cf23bbe
Nov 03 00:45:38.624394 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:38.624345    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql" event={"ID":"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae","Type":"ContainerStarted","Data":"8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be"}
Nov 03 00:45:38.624911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:38.624457    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:38.624911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:38.624487    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql" event={"ID":"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae","Type":"ContainerStarted","Data":"a33aa79688a40de926a63f738569542dee1c1be6f5f287fe1de7db9c3cf23bbe"}
Nov 03 00:45:38.662056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:38.662002    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql" podStartSLOduration=1.661984553 podStartE2EDuration="1.661984553s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:38.661563233 +0000 UTC m=+2376.216915258" watchObservedRunningTime="2026-11-03 00:45:38.661984553 +0000 UTC m=+2376.217336577"
Nov 03 00:45:41.569282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:41.568826    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:45:41.569282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:41.568994    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:45:41.918175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:41.916292    6000 patch_prober.go:29] interesting pod/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:45:41.918175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:41.916368    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:45:43.075715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.075490    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:45:43.118017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.117987    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-credential-socket\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118039    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-socket\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118066    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-podinfo\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118101    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-certs\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118204    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-kube-api-access-tzfng\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-kube-api-access-tzfng\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118305    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df7a4329-145a-41c8-bc2d-97dd98a8d113-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7a4329-145a-41c8-bc2d-97dd98a8d113-istiod-ca-cert\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118421    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-data\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118474    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-envoy\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.118712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118508    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-token\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-token\") pod \"df7a4329-145a-41c8-bc2d-97dd98a8d113\" (UID: \"df7a4329-145a-41c8-bc2d-97dd98a8d113\") "
Nov 03 00:45:43.122543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.118954    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-credential-socket" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.122543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.119321    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.122543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.119833    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df7a4329-145a-41c8-bc2d-97dd98a8d113-istiod-ca-cert" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:43.122543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.119888    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-certs" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.122543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.120079    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-data" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.122543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.120130    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-socket" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.122543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.122442    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-token" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:43.123001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.122753    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-podinfo" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:45:43.123001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.122939    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-kube-api-access-tzfng" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "kube-api-access-tzfng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:43.132308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.129394    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-envoy" pod "df7a4329-145a-41c8-bc2d-97dd98a8d113" (UID: "df7a4329-145a-41c8-bc2d-97dd98a8d113"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.185701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.185679    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:45:43.220374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220345    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-socket\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.220510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220390    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-token\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-token\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.220510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220455    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-credential-socket\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.220510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220497    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istiod-ca-cert\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.220645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220524    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-certs\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.220645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220604    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-envoy\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.220645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220629    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-socket" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.220824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220795    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-credential-socket" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.220947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220890    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-kube-api-access-gm6qk\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-kube-api-access-gm6qk\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.221137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220951    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-podinfo\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.221137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220973    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istiod-ca-cert" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:43.221137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.220980    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-certs" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.221137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221042    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-data\") pod \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\" (UID: \"8e4dfcbe-7183-4a23-87b7-8da7f51823a7\") "
Nov 03 00:45:43.221489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221456    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-data" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.221889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221865    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzfng\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-kube-api-access-tzfng\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221894    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/df7a4329-145a-41c8-bc2d-97dd98a8d113-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221909    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221921    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221934    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221947    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221960    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221972    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221985    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.221999    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.222000 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.222012    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.223640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.222026    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/df7a4329-145a-41c8-bc2d-97dd98a8d113-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.223640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.222042    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/df7a4329-145a-41c8-bc2d-97dd98a8d113-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.224196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.224161    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-kube-api-access-gm6qk" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "kube-api-access-gm6qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:43.225381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.224504    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-podinfo" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:45:43.225381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.225368    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-token" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:43.226020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.225954    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-envoy" pod "8e4dfcbe-7183-4a23-87b7-8da7f51823a7" (UID: "8e4dfcbe-7183-4a23-87b7-8da7f51823a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:43.323670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.323589    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gm6qk\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-kube-api-access-gm6qk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.323670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.323617    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.323670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.323631    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.323670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.323643    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e4dfcbe-7183-4a23-87b7-8da7f51823a7-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:43.700991 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.700958    6000 generic.go:356] "Generic (PLEG): container finished" podID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerID="b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790" exitCode=0
Nov 03 00:45:43.701217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.701033    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"
Nov 03 00:45:43.701217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.701053    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerDied","Data":"b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790"}
Nov 03 00:45:43.701217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.701157    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch" event={"ID":"8e4dfcbe-7183-4a23-87b7-8da7f51823a7","Type":"ContainerDied","Data":"fc1b222f7f9e4d1bb90d9ddffeb32eb14a8a5dfeb3ee46132461e47790b1d9c7"}
Nov 03 00:45:43.701217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.701189    6000 scope.go:122] "RemoveContainer" containerID="b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790"
Nov 03 00:45:43.708293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.704547    6000 generic.go:356] "Generic (PLEG): container finished" podID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerID="8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91" exitCode=0
Nov 03 00:45:43.708293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.704695    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" event={"ID":"df7a4329-145a-41c8-bc2d-97dd98a8d113","Type":"ContainerDied","Data":"8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91"}
Nov 03 00:45:43.708293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.704746    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz" event={"ID":"df7a4329-145a-41c8-bc2d-97dd98a8d113","Type":"ContainerDied","Data":"52c8e690d0b5b9fb792218c47b02624a0926ae1547c55c302b72b4e46de75c07"}
Nov 03 00:45:43.708293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.704832    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"
Nov 03 00:45:43.731904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.731882    6000 scope.go:122] "RemoveContainer" containerID="4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537"
Nov 03 00:45:43.742223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.742203    6000 scope.go:122] "RemoveContainer" containerID="b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790"
Nov 03 00:45:43.742503 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:43.742486    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790\": container with ID starting with b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790 not found: ID does not exist" containerID="b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790"
Nov 03 00:45:43.742551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.742512    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790"} err="failed to get container status \"b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790\": rpc error: code = NotFound desc = could not find container \"b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790\": container with ID starting with b68e4bac6227950c27911dc60fa9a51b1f0f712aeb692413b4e68dab1dfaa790 not found: ID does not exist"
Nov 03 00:45:43.742551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.742535    6000 scope.go:122] "RemoveContainer" containerID="4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537"
Nov 03 00:45:43.742789 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:43.742765    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537\": container with ID starting with 4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537 not found: ID does not exist" containerID="4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537"
Nov 03 00:45:43.742874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.742799    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537"} err="failed to get container status \"4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537\": rpc error: code = NotFound desc = could not find container \"4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537\": container with ID starting with 4c2d240d6271d10a5268b4da9e701580fe0ad09a4a5ee8ac0b9fc949c18d8537 not found: ID does not exist"
Nov 03 00:45:43.742874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.742822    6000 scope.go:122] "RemoveContainer" containerID="8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91"
Nov 03 00:45:43.752542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.752523    6000 scope.go:122] "RemoveContainer" containerID="8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91"
Nov 03 00:45:43.752781 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:43.752761    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91\": container with ID starting with 8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91 not found: ID does not exist" containerID="8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91"
Nov 03 00:45:43.752847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.752792    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91"} err="failed to get container status \"8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91\": rpc error: code = NotFound desc = could not find container \"8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91\": container with ID starting with 8a25d8500702afacf826cc883a474e5329415baa051631f61d3723ee80cf4f91 not found: ID does not exist"
Nov 03 00:45:43.767325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.767296    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"]
Nov 03 00:45:43.778213 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.778192    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-qjtjz"]
Nov 03 00:45:43.804857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.804831    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"]
Nov 03 00:45:43.808569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:43.808544    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-lgsch"]
Nov 03 00:45:44.568898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:44.568867    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" path="/var/lib/kubelet/pods/8e4dfcbe-7183-4a23-87b7-8da7f51823a7/volumes"
Nov 03 00:45:44.569583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:44.569567    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" path="/var/lib/kubelet/pods/df7a4329-145a-41c8-bc2d-97dd98a8d113/volumes"
Nov 03 00:45:49.375796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.375760    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"]
Nov 03 00:45:49.466384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.466352    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"]
Nov 03 00:45:49.467167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467149    6000 cpu_manager.go:414] "containerMap: removing container" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:49.467167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467167    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467180    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467185    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ea3b26-88dc-48fb-844d-1d37365e9e3d" containerName="cleanup"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467197    6000 cpu_manager.go:414] "containerMap: removing container" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467204    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="777d72c8-be3c-49a9-8841-dc8aa04eecea" containerName="cleanup"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467222    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467227    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467238    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467242    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467269    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467275    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f000b77c-fc1d-4f65-b7be-b4a0b0814ac3" containerName="cleanup"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467284    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467289    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467304    6000 cpu_manager.go:414] "containerMap: removing container" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy"
Nov 03 00:45:49.467358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467308    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy"
Nov 03 00:45:49.467931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467438    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.467931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467451    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="df7a4329-145a-41c8-bc2d-97dd98a8d113" containerName="istio-proxy"
Nov 03 00:45:49.467931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467464    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1db35fa-8e4b-4908-9131-889f5efaf085" containerName="cleanup"
Nov 03 00:45:49.467931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467475    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="df147ebf-b21f-4df2-9e5d-05bcb412ca53" containerName="cleanup"
Nov 03 00:45:49.467931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.467481    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:45:49.472852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.472834    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.490468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.490434    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-fdacdb-openshift-default-dockercfg-2krd9\""
Nov 03 00:45:49.494049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.494019    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"]
Nov 03 00:45:49.515509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515468    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.515699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515600    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.515699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515683    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.515817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515711    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.515817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515808    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.515935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515837    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.515935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515864    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.516037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515930    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz9wl\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-kube-api-access-hz9wl\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.516037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.515997    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.617761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.617727    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.617951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.617821    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.617951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.617849    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.617953    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.617982    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618229 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618206    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618282    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618356    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618363    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz9wl\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-kube-api-access-hz9wl\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618459    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618580    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618742    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618810    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.618990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.618969    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.620780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.620754    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.621031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.621009    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.637437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.637357    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:45:49.637771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.637745    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.638107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.638087    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz9wl\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-kube-api-access-hz9wl\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.709102 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.709064    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"]
Nov 03 00:45:49.797065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:49.797034    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:49.907505 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:49.907417    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:45:50.122110 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:45:50.122079    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5f12ebd_dce0_4c4e_920b_c260acdb3b17.slice/crio-46d9901db10d4fbe2893fd5a95f4eda33fb3aa1b17d409a3de4161e163a34adc WatchSource:0}: Error finding container 46d9901db10d4fbe2893fd5a95f4eda33fb3aa1b17d409a3de4161e163a34adc: Status 404 returned error can't find the container with id 46d9901db10d4fbe2893fd5a95f4eda33fb3aa1b17d409a3de4161e163a34adc
Nov 03 00:45:50.122462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.122441    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"]
Nov 03 00:45:50.125060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.125030    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:45:50.125141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.125095    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:45:50.125141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.125122    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:45:50.763670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.763628    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" event={"ID":"a5f12ebd-dce0-4c4e-920b-c260acdb3b17","Type":"ContainerStarted","Data":"b45e01494524204cec11cb1362db77f3e2d0efe0b86848fb09c50dcaa9657859"}
Nov 03 00:45:50.764194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.763687    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" event={"ID":"a5f12ebd-dce0-4c4e-920b-c260acdb3b17","Type":"ContainerStarted","Data":"46d9901db10d4fbe2893fd5a95f4eda33fb3aa1b17d409a3de4161e163a34adc"}
Nov 03 00:45:50.797748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.797703    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:50.804669 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:50.804612    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podStartSLOduration=1.80459625 podStartE2EDuration="1.80459625s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:45:50.803710393 +0000 UTC m=+2388.359062427" watchObservedRunningTime="2026-11-03 00:45:50.80459625 +0000 UTC m=+2388.359948272"
Nov 03 00:45:51.812881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:51.812848    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:45:51.813344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:51.812903    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 00:45:52.808280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:52.807782    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:45:52.808280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:52.807868    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 00:45:52.936223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:52.936187    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"]
Nov 03 00:45:52.936662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:52.936429    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy" containerID="cri-o://f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02" gracePeriod=30
Nov 03 00:45:53.807282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:53.805703    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:45:53.807282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:53.805778    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 00:45:54.376821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.376759    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l" podUID="53ae069a-30d9-4b36-ac0a-1d4d21f272cd" containerName="payload-processing" containerID="cri-o://72c441ae1e8c43d7bac021e54c98868caa36b83601b832ede5eb4b845253d5cf" gracePeriod=25
Nov 03 00:45:54.718009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.717319    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c" podUID="373bc38c-e76a-4f1d-a73a-9763d466f097" containerName="payload-processing" containerID="cri-o://6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6" gracePeriod=25
Nov 03 00:45:54.803360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.800685    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:45:54.803360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.800751    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 00:45:54.820836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.820808    6000 generic.go:356] "Generic (PLEG): container finished" podID="53ae069a-30d9-4b36-ac0a-1d4d21f272cd" containerID="72c441ae1e8c43d7bac021e54c98868caa36b83601b832ede5eb4b845253d5cf" exitCode=0
Nov 03 00:45:54.820997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.820889    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l" event={"ID":"53ae069a-30d9-4b36-ac0a-1d4d21f272cd","Type":"ContainerDied","Data":"72c441ae1e8c43d7bac021e54c98868caa36b83601b832ede5eb4b845253d5cf"}
Nov 03 00:45:54.861290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.861239    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:45:54.938295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.937732    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-plugins-config-volume\") pod \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") "
Nov 03 00:45:54.938295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.938154    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-plugins-config-volume" pod "53ae069a-30d9-4b36-ac0a-1d4d21f272cd" (UID: "53ae069a-30d9-4b36-ac0a-1d4d21f272cd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:54.938295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.938212    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-kube-api-access-c849m\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-kube-api-access-c849m\") pod \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") "
Nov 03 00:45:54.938542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.938346    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-provider-credentials\") pod \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") "
Nov 03 00:45:54.938542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.938395    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-routing-overlay-volume\") pod \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\" (UID: \"53ae069a-30d9-4b36-ac0a-1d4d21f272cd\") "
Nov 03 00:45:54.939021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.938999    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:54.939444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.939400    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-routing-overlay-volume" pod "53ae069a-30d9-4b36-ac0a-1d4d21f272cd" (UID: "53ae069a-30d9-4b36-ac0a-1d4d21f272cd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:54.942118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.942092    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-kube-api-access-c849m" pod "53ae069a-30d9-4b36-ac0a-1d4d21f272cd" (UID: "53ae069a-30d9-4b36-ac0a-1d4d21f272cd"). InnerVolumeSpecName "kube-api-access-c849m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:54.942553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:54.942529    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-provider-credentials" pod "53ae069a-30d9-4b36-ac0a-1d4d21f272cd" (UID: "53ae069a-30d9-4b36-ac0a-1d4d21f272cd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:55.040408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.040383    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c849m\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-kube-api-access-c849m\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:55.040591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.040577    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:55.040673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.040661    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/53ae069a-30d9-4b36-ac0a-1d4d21f272cd-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:55.045038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.045017    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:45:55.141850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.141817    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-kube-api-access-vzj6b\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-kube-api-access-vzj6b\") pod \"373bc38c-e76a-4f1d-a73a-9763d466f097\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") "
Nov 03 00:45:55.142070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.141966    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-plugins-config-volume\") pod \"373bc38c-e76a-4f1d-a73a-9763d466f097\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") "
Nov 03 00:45:55.142150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.142084    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-provider-credentials\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-provider-credentials\") pod \"373bc38c-e76a-4f1d-a73a-9763d466f097\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") "
Nov 03 00:45:55.142417 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.142390    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-routing-overlay-volume\") pod \"373bc38c-e76a-4f1d-a73a-9763d466f097\" (UID: \"373bc38c-e76a-4f1d-a73a-9763d466f097\") "
Nov 03 00:45:55.142803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.142776    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-plugins-config-volume" pod "373bc38c-e76a-4f1d-a73a-9763d466f097" (UID: "373bc38c-e76a-4f1d-a73a-9763d466f097"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:55.143096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.143074    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-routing-overlay-volume" pod "373bc38c-e76a-4f1d-a73a-9763d466f097" (UID: "373bc38c-e76a-4f1d-a73a-9763d466f097"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:55.143378 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.143357    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:55.143505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.143491    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373bc38c-e76a-4f1d-a73a-9763d466f097-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:55.145484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.145459    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-kube-api-access-vzj6b" pod "373bc38c-e76a-4f1d-a73a-9763d466f097" (UID: "373bc38c-e76a-4f1d-a73a-9763d466f097"). InnerVolumeSpecName "kube-api-access-vzj6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:55.145588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.145488    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-provider-credentials" pod "373bc38c-e76a-4f1d-a73a-9763d466f097" (UID: "373bc38c-e76a-4f1d-a73a-9763d466f097"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:55.245104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.245022    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzj6b\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-kube-api-access-vzj6b\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:55.245104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.245053    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373bc38c-e76a-4f1d-a73a-9763d466f097-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:55.810281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.809655    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 03 00:45:55.810281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.809718    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 03 00:45:55.829697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.829664    6000 generic.go:356] "Generic (PLEG): container finished" podID="373bc38c-e76a-4f1d-a73a-9763d466f097" containerID="6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6" exitCode=0
Nov 03 00:45:55.829852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.829719    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"
Nov 03 00:45:55.829852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.829745    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c" event={"ID":"373bc38c-e76a-4f1d-a73a-9763d466f097","Type":"ContainerDied","Data":"6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6"}
Nov 03 00:45:55.829852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.829799    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c" event={"ID":"373bc38c-e76a-4f1d-a73a-9763d466f097","Type":"ContainerDied","Data":"cc314bc896477d869aef711fd8711be7b9ced6f4979dcf8984e29ff57d2432f6"}
Nov 03 00:45:55.829852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.829821    6000 scope.go:122] "RemoveContainer" containerID="6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6"
Nov 03 00:45:55.831553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.831532    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"
Nov 03 00:45:55.831695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.831573    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l" event={"ID":"53ae069a-30d9-4b36-ac0a-1d4d21f272cd","Type":"ContainerDied","Data":"4f71b57f38920aade6443d10635f23a4624a2190fcf27b4c978aae41f13b458f"}
Nov 03 00:45:55.842534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.842512    6000 scope.go:122] "RemoveContainer" containerID="6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6"
Nov 03 00:45:55.842824 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:55.842803    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6\": container with ID starting with 6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6 not found: ID does not exist" containerID="6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6"
Nov 03 00:45:55.842903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.842832    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6"} err="failed to get container status \"6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6\": rpc error: code = NotFound desc = could not find container \"6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6\": container with ID starting with 6cd6dd6d9bee4fff13506a6f5d86679dd680e5db5289119b20327966943f0bf6 not found: ID does not exist"
Nov 03 00:45:55.842903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.842849    6000 scope.go:122] "RemoveContainer" containerID="72c441ae1e8c43d7bac021e54c98868caa36b83601b832ede5eb4b845253d5cf"
Nov 03 00:45:55.880879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.880847    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"]
Nov 03 00:45:55.915964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.915931    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999nh29l"]
Nov 03 00:45:55.933548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.933517    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"]
Nov 03 00:45:55.937985 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:55.937959    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65cc7bk5c"]
Nov 03 00:45:56.572773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:56.572729    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373bc38c-e76a-4f1d-a73a-9763d466f097" path="/var/lib/kubelet/pods/373bc38c-e76a-4f1d-a73a-9763d466f097/volumes"
Nov 03 00:45:56.573552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:56.573524    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53ae069a-30d9-4b36-ac0a-1d4d21f272cd" path="/var/lib/kubelet/pods/53ae069a-30d9-4b36-ac0a-1d4d21f272cd/volumes"
Nov 03 00:45:56.851592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:56.851440    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:56.852184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:56.851884    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:56.852713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:56.852687    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 00:45:58.215571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.215503    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:45:58.301370 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301337    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-data\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301415    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-istio-token\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-istio-token\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301467    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-envoy\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301498    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/797b5565-9c2a-4dbd-9f73-20865f557797-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797b5565-9c2a-4dbd-9f73-20865f557797-istio-podinfo\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301553    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-socket\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301587    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-certs\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301683    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-credential-socket\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301747    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/797b5565-9c2a-4dbd-9f73-20865f557797-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797b5565-9c2a-4dbd-9f73-20865f557797-istiod-ca-cert\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.301944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301699    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-data" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:58.301944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.301922    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-kube-api-access-wczxd\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-kube-api-access-wczxd\") pod \"797b5565-9c2a-4dbd-9f73-20865f557797\" (UID: \"797b5565-9c2a-4dbd-9f73-20865f557797\") "
Nov 03 00:45:58.302051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302003    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-certs" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:58.302154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302130    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-socket" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:58.302217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302201    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-credential-socket" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:58.302742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302583    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.302742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302611    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.302742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302628    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.302742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302641    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.302742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.302693    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/797b5565-9c2a-4dbd-9f73-20865f557797-istiod-ca-cert" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:45:58.304687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.304657    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-envoy" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:45:58.304814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.304793    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-istio-token" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:58.305430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.305399    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-kube-api-access-wczxd" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "kube-api-access-wczxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:45:58.305608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.305561    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/797b5565-9c2a-4dbd-9f73-20865f557797-istio-podinfo" pod "797b5565-9c2a-4dbd-9f73-20865f557797" (UID: "797b5565-9c2a-4dbd-9f73-20865f557797"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:45:58.404485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.404450    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/797b5565-9c2a-4dbd-9f73-20865f557797-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.404485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.404483    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczxd\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-kube-api-access-wczxd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.404697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.404497    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/797b5565-9c2a-4dbd-9f73-20865f557797-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.404697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.404505    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/797b5565-9c2a-4dbd-9f73-20865f557797-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.404697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.404514    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/797b5565-9c2a-4dbd-9f73-20865f557797-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:45:58.860550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.860517    6000 generic.go:356] "Generic (PLEG): container finished" podID="797b5565-9c2a-4dbd-9f73-20865f557797" containerID="f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02" exitCode=0
Nov 03 00:45:58.860753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.860567    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"
Nov 03 00:45:58.860753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.860600    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" event={"ID":"797b5565-9c2a-4dbd-9f73-20865f557797","Type":"ContainerDied","Data":"f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02"}
Nov 03 00:45:58.860753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.860660    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz" event={"ID":"797b5565-9c2a-4dbd-9f73-20865f557797","Type":"ContainerDied","Data":"ef596565d65bdb1bb3261bf1d0126b557cb1870f8c4530bd0755b33bb83b4278"}
Nov 03 00:45:58.860753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.860692    6000 scope.go:122] "RemoveContainer" containerID="f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02"
Nov 03 00:45:58.872675 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.872653    6000 scope.go:122] "RemoveContainer" containerID="f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02"
Nov 03 00:45:58.873001 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:45:58.872969    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02\": container with ID starting with f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02 not found: ID does not exist" containerID="f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02"
Nov 03 00:45:58.873058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.873011    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02"} err="failed to get container status \"f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02\": rpc error: code = NotFound desc = could not find container \"f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02\": container with ID starting with f35fe8600d66c1c5ea57bb3ec692887f200a90e0e05ece3fbaa28cec29af8b02 not found: ID does not exist"
Nov 03 00:45:58.924683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.924647    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"]
Nov 03 00:45:58.927717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:45:58.927697    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-cd6bz"]
Nov 03 00:46:00.568604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:00.568574    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" path="/var/lib/kubelet/pods/797b5565-9c2a-4dbd-9f73-20865f557797/volumes"
Nov 03 00:46:14.214162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.214130    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"]
Nov 03 00:46:14.215017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.214997    6000 cpu_manager.go:414] "containerMap: removing container" podUID="53ae069a-30d9-4b36-ac0a-1d4d21f272cd" containerName="payload-processing"
Nov 03 00:46:14.215017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215019    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53ae069a-30d9-4b36-ac0a-1d4d21f272cd" containerName="payload-processing"
Nov 03 00:46:14.215170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215037    6000 cpu_manager.go:414] "containerMap: removing container" podUID="373bc38c-e76a-4f1d-a73a-9763d466f097" containerName="payload-processing"
Nov 03 00:46:14.215170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215045    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373bc38c-e76a-4f1d-a73a-9763d466f097" containerName="payload-processing"
Nov 03 00:46:14.215170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215057    6000 cpu_manager.go:414] "containerMap: removing container" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy"
Nov 03 00:46:14.215170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215063    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy"
Nov 03 00:46:14.215320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215226    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="373bc38c-e76a-4f1d-a73a-9763d466f097" containerName="payload-processing"
Nov 03 00:46:14.215320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215235    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="797b5565-9c2a-4dbd-9f73-20865f557797" containerName="istio-proxy"
Nov 03 00:46:14.215320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215249    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e4dfcbe-7183-4a23-87b7-8da7f51823a7" containerName="istio-proxy"
Nov 03 00:46:14.215320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.215280    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="53ae069a-30d9-4b36-ac0a-1d4d21f272cd" containerName="payload-processing"
Nov 03 00:46:14.220938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.220916    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.241841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.241807    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"]
Nov 03 00:46:14.249402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.249371    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"e2e-cmp-legacy-31cedc-provider-cred\""
Nov 03 00:46:14.249545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.249477    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"openshift-service-ca.crt\""
Nov 03 00:46:14.249636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.249613    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"routing-overlay\""
Nov 03 00:46:14.249815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.249794    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"payload-processing-external-model-e2e--bf641009-dockercfg-lh4nm\""
Nov 03 00:46:14.250082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.250065    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"payload-processing-external-model-plugins-e2e-extmod-a-c738e0\""
Nov 03 00:46:14.250208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.249880    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"kube-root-ca.crt\""
Nov 03 00:46:14.405279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.404662    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.405279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.404835    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.405279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.404896    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc54v\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-kube-api-access-bc54v\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.405279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.404940    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.506575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.506468    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.506575 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.506547    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc54v\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-kube-api-access-bc54v\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.506829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.506589    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.506829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.506791    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.508735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.508710    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.508966 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.508927    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.511149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.511128    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.524986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.524966    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc54v\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-kube-api-access-bc54v\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.535924 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.535900    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:14.845442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:14.845412    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"]
Nov 03 00:46:14.845578 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:46:14.845535    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04d2687_9948_4422_8eb7_57ed70de723c.slice/crio-f888e2c3de1c2a2d32021783863cd30abfbf7d03ebd7f46bd72a074c2142f5f7 WatchSource:0}: Error finding container f888e2c3de1c2a2d32021783863cd30abfbf7d03ebd7f46bd72a074c2142f5f7: Status 404 returned error can't find the container with id f888e2c3de1c2a2d32021783863cd30abfbf7d03ebd7f46bd72a074c2142f5f7
Nov 03 00:46:15.021599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:15.021505    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t" event={"ID":"c04d2687-9948-4422-8eb7-57ed70de723c","Type":"ContainerStarted","Data":"cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55"}
Nov 03 00:46:15.021785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:15.021619    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t" event={"ID":"c04d2687-9948-4422-8eb7-57ed70de723c","Type":"ContainerStarted","Data":"f888e2c3de1c2a2d32021783863cd30abfbf7d03ebd7f46bd72a074c2142f5f7"}
Nov 03 00:46:15.021785 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:15.021658    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:15.060067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:15.060017    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t" podStartSLOduration=1.059998102 podStartE2EDuration="1.059998102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:15.059499267 +0000 UTC m=+2412.614851286" watchObservedRunningTime="2026-11-03 00:46:15.059998102 +0000 UTC m=+2412.615350127"
Nov 03 00:46:19.900410 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:46:19.900372    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:46:26.032650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:26.032619    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:46:28.470532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:28.470492    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:46:28.988184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:28.988151    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"]
Nov 03 00:46:28.998418 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:28.998384    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.011824 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.011794    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"]
Nov 03 00:46:29.012573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.012545    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"agc-cred-29e6aeee329d\""
Nov 03 00:46:29.012707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.012666    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"openshift-service-ca.crt\""
Nov 03 00:46:29.012774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.012706    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"routing-overlay\""
Nov 03 00:46:29.012923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.012901    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"payload-processing-external-model-e2e--0fa13bfc-dockercfg-w5lhh\""
Nov 03 00:46:29.018477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.018455    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ac338d\""
Nov 03 00:46:29.018477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.018460    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"kube-root-ca.crt\""
Nov 03 00:46:29.197975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.197936    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.198198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.198055    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlv2\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-kube-api-access-7nlv2\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.198693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.198665    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.198831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.198716    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.300701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.300624    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.300701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.300671    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.300957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.300713    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.300957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.300782    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlv2\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-kube-api-access-7nlv2\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.301458 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.301423    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.301796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.301519    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.303534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.303513    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.319582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.319556    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlv2\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-kube-api-access-7nlv2\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:29.630279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:29.626604    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:30.161448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:30.161419    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"]
Nov 03 00:46:30.176536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:30.176478    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4" event={"ID":"8be5dc52-85ed-4c07-8b3b-8549fc818b9e","Type":"ContainerStarted","Data":"6926587d1502350f57839cd5feb89473dd3c8d770159747e5ab9394ce04420af"}
Nov 03 00:46:31.186080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:31.186041    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4" event={"ID":"8be5dc52-85ed-4c07-8b3b-8549fc818b9e","Type":"ContainerStarted","Data":"c48be655b4f0fadd689b7f9d0c5305de512a3a92fa2c327fd135807a4058d12f"}
Nov 03 00:46:31.186577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:31.186126    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:31.229210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:31.229150    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4" podStartSLOduration=3.229133132 podStartE2EDuration="3.229133132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:31.226620974 +0000 UTC m=+2428.781972998" watchObservedRunningTime="2026-11-03 00:46:31.229133132 +0000 UTC m=+2428.784485158"
Nov 03 00:46:33.859136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.859104    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"]
Nov 03 00:46:33.864044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.864026    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.879175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.879152    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"agc-cred-6ab1a9c6ac02\""
Nov 03 00:46:33.883121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.883099    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"]
Nov 03 00:46:33.895416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.895383    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.895688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.895662    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.895789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.895744    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xqjx\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-kube-api-access-6xqjx\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.895851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.895802    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.997568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.997529    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.997792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.997620    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xqjx\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-kube-api-access-6xqjx\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.997792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.997674    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.997914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.997793    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.998459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.998427    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:33.998577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:33.998536    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:34.001122 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:34.001095    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:34.019355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:34.019324    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xqjx\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-kube-api-access-6xqjx\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:34.181597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:34.181510    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:34.499448 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:46:34.499411    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc1f9283_5f2a_4b95_a41d_2622a69c8095.slice/crio-3155b8b2487d582f1b62e2b247a672b4733ca442a228fab1b40560ce23d0347c WatchSource:0}: Error finding container 3155b8b2487d582f1b62e2b247a672b4733ca442a228fab1b40560ce23d0347c: Status 404 returned error can't find the container with id 3155b8b2487d582f1b62e2b247a672b4733ca442a228fab1b40560ce23d0347c
Nov 03 00:46:34.500541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:34.500509    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"]
Nov 03 00:46:35.234061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:35.234025    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl" event={"ID":"dc1f9283-5f2a-4b95-a41d-2622a69c8095","Type":"ContainerStarted","Data":"26dc4c458108c09a0252cec6e582764e8f67485dc60bdd4a5a9e30aaea3e7147"}
Nov 03 00:46:35.234547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:35.234081    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl" event={"ID":"dc1f9283-5f2a-4b95-a41d-2622a69c8095","Type":"ContainerStarted","Data":"3155b8b2487d582f1b62e2b247a672b4733ca442a228fab1b40560ce23d0347c"}
Nov 03 00:46:35.234547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:35.234108    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:35.274104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:35.274042    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl" podStartSLOduration=2.274026681 podStartE2EDuration="2.274026681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:35.270475165 +0000 UTC m=+2432.825827188" watchObservedRunningTime="2026-11-03 00:46:35.274026681 +0000 UTC m=+2432.829378711"
Nov 03 00:46:37.414441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.414400    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"]
Nov 03 00:46:37.414961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.414685    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy" containerID="cri-o://28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647" gracePeriod=30
Nov 03 00:46:37.464607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.464572    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"]
Nov 03 00:46:37.469880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.469857    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.484342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.484314    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdacdb\""
Nov 03 00:46:37.484486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.484397    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"kube-root-ca.crt\""
Nov 03 00:46:37.484486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.484451    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"openshift-service-ca.crt\""
Nov 03 00:46:37.486796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.486768    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"]
Nov 03 00:46:37.489307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.489286    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"routing-overlay\""
Nov 03 00:46:37.508300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.504801    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-2274efc91ed5\""
Nov 03 00:46:37.508300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.504889    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-536b490d5ab1\""
Nov 03 00:46:37.508300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.504913    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-e2e--2e23f326-dockercfg-njdlx\""
Nov 03 00:46:37.559372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.559336    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.559561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.559439    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.559628 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.559612    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbgg4\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-kube-api-access-gbgg4\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.559667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.559640    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.660893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.660858    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbgg4\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-kube-api-access-gbgg4\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.661099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.660909    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.661181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.661113    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.661181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.661153    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.661938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.661912    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.662076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.662056    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.663803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.663785    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.679487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.679428    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbgg4\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-kube-api-access-gbgg4\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:37.789023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:37.788987    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:38.095345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:38.095313    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"]
Nov 03 00:46:38.095932 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:46:38.095911    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0655e7a3_f738_4b56_80f9_1363f8b75081.slice/crio-a77a8caf538fced2850e58a36eb2c2a9f1dac27e03ec026db2ebb8e7c5cf9bbc WatchSource:0}: Error finding container a77a8caf538fced2850e58a36eb2c2a9f1dac27e03ec026db2ebb8e7c5cf9bbc: Status 404 returned error can't find the container with id a77a8caf538fced2850e58a36eb2c2a9f1dac27e03ec026db2ebb8e7c5cf9bbc
Nov 03 00:46:38.270659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:38.270624    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57" event={"ID":"0655e7a3-f738-4b56-80f9-1363f8b75081","Type":"ContainerStarted","Data":"bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31"}
Nov 03 00:46:38.270861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:38.270692    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57" event={"ID":"0655e7a3-f738-4b56-80f9-1363f8b75081","Type":"ContainerStarted","Data":"a77a8caf538fced2850e58a36eb2c2a9f1dac27e03ec026db2ebb8e7c5cf9bbc"}
Nov 03 00:46:38.270861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:38.270823    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:38.312524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:38.312454    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57" podStartSLOduration=1.312433494 podStartE2EDuration="1.312433494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:46:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:46:38.310120453 +0000 UTC m=+2435.865472487" watchObservedRunningTime="2026-11-03 00:46:38.312433494 +0000 UTC m=+2435.867785524"
Nov 03 00:46:42.204943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.204290    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:46:42.699635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.699606    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:46:42.733547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733519    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-kube-api-access-mst2c\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-kube-api-access-mst2c\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.733683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733612    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68d5d410-6702-46d1-a442-6d712cd0705f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5d410-6702-46d1-a442-6d712cd0705f-istiod-ca-cert\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.733683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733671    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-socket\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.733777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733721    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-data\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.733777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733756    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-certs\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.733898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733808    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-istio-token\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-istio-token\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.733898 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733859    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-envoy\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.734006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733916    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-credential-socket\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.734063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.733995    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-socket" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:42.734063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734025    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-data" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:42.734063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734053    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-certs" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:42.734196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734069    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/68d5d410-6702-46d1-a442-6d712cd0705f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5d410-6702-46d1-a442-6d712cd0705f-istio-podinfo\") pod \"68d5d410-6702-46d1-a442-6d712cd0705f\" (UID: \"68d5d410-6702-46d1-a442-6d712cd0705f\") "
Nov 03 00:46:42.734196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734073    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68d5d410-6702-46d1-a442-6d712cd0705f-istiod-ca-cert" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:46:42.734314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734226    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-credential-socket" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:42.734732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734700    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/68d5d410-6702-46d1-a442-6d712cd0705f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.734732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734722    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.734914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734735    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.734914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734748    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.734914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.734760    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.735972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.735953    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-kube-api-access-mst2c" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "kube-api-access-mst2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:42.736494 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.736469    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/68d5d410-6702-46d1-a442-6d712cd0705f-istio-podinfo" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:46:42.736716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.736697    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-envoy" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:46:42.736716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.736709    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-istio-token" pod "68d5d410-6702-46d1-a442-6d712cd0705f" (UID: "68d5d410-6702-46d1-a442-6d712cd0705f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:42.836250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.836160    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.836250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.836195    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/68d5d410-6702-46d1-a442-6d712cd0705f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.836250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.836204    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/68d5d410-6702-46d1-a442-6d712cd0705f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:42.836250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:42.836215    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mst2c\" (UniqueName: \"kubernetes.io/projected/68d5d410-6702-46d1-a442-6d712cd0705f-kube-api-access-mst2c\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:43.311203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.311169    6000 generic.go:356] "Generic (PLEG): container finished" podID="68d5d410-6702-46d1-a442-6d712cd0705f" containerID="28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647" exitCode=0
Nov 03 00:46:43.311662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.311219    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"
Nov 03 00:46:43.311662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.311243    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" event={"ID":"68d5d410-6702-46d1-a442-6d712cd0705f","Type":"ContainerDied","Data":"28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647"}
Nov 03 00:46:43.311662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.311300    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn" event={"ID":"68d5d410-6702-46d1-a442-6d712cd0705f","Type":"ContainerDied","Data":"1b46031adaf42707fbae98ff42bba8bad8a8dc4e98c3820c75010da6a71fd2ea"}
Nov 03 00:46:43.311662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.311324    6000 scope.go:122] "RemoveContainer" containerID="28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647"
Nov 03 00:46:43.327639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.327620    6000 scope.go:122] "RemoveContainer" containerID="28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647"
Nov 03 00:46:43.327927 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:46:43.327908    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647\": container with ID starting with 28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647 not found: ID does not exist" containerID="28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647"
Nov 03 00:46:43.327988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.327935    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647"} err="failed to get container status \"28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647\": rpc error: code = NotFound desc = could not find container \"28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647\": container with ID starting with 28efedd1d58c00d67e3e7ad7f0627771618dd83319afeead4e4ddff5008ac647 not found: ID does not exist"
Nov 03 00:46:43.374777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.374746    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"]
Nov 03 00:46:43.377853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:43.377828    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-26rwn"]
Nov 03 00:46:44.568157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:44.568123    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" path="/var/lib/kubelet/pods/68d5d410-6702-46d1-a442-6d712cd0705f/volumes"
Nov 03 00:46:46.244523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:46.244490    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:46:46.315072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:46.315037    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"]
Nov 03 00:46:49.281380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:49.281350    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:46:49.898130 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:46:49.898089    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:46:51.315749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.315711    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf" podUID="f98f73ad-617e-4322-b81f-60bef48a13be" containerName="payload-processing" containerID="cri-o://88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76" gracePeriod=25
Nov 03 00:46:51.594020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.593993    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:46:51.679895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.679843    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-plugins-config-volume\") pod \"f98f73ad-617e-4322-b81f-60bef48a13be\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") "
Nov 03 00:46:51.680309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.680154    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-kube-api-access-8jkd4\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-kube-api-access-8jkd4\") pod \"f98f73ad-617e-4322-b81f-60bef48a13be\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") "
Nov 03 00:46:51.680309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.680221    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-routing-overlay-volume\") pod \"f98f73ad-617e-4322-b81f-60bef48a13be\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") "
Nov 03 00:46:51.680309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.680302    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-provider-credentials\") pod \"f98f73ad-617e-4322-b81f-60bef48a13be\" (UID: \"f98f73ad-617e-4322-b81f-60bef48a13be\") "
Nov 03 00:46:51.680639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.680614    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-plugins-config-volume" pod "f98f73ad-617e-4322-b81f-60bef48a13be" (UID: "f98f73ad-617e-4322-b81f-60bef48a13be"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:46:51.680897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.680881    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-routing-overlay-volume" pod "f98f73ad-617e-4322-b81f-60bef48a13be" (UID: "f98f73ad-617e-4322-b81f-60bef48a13be"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:46:51.681041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.681029    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:51.681080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.681044    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f98f73ad-617e-4322-b81f-60bef48a13be-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:51.683590 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.683566    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-provider-credentials" pod "f98f73ad-617e-4322-b81f-60bef48a13be" (UID: "f98f73ad-617e-4322-b81f-60bef48a13be"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:51.683704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.683586    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-kube-api-access-8jkd4" pod "f98f73ad-617e-4322-b81f-60bef48a13be" (UID: "f98f73ad-617e-4322-b81f-60bef48a13be"). InnerVolumeSpecName "kube-api-access-8jkd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:46:51.782064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.782032    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jkd4\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-kube-api-access-8jkd4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:51.782064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:51.782060    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f98f73ad-617e-4322-b81f-60bef48a13be-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:46:52.381481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.381448    6000 generic.go:356] "Generic (PLEG): container finished" podID="f98f73ad-617e-4322-b81f-60bef48a13be" containerID="88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76" exitCode=0
Nov 03 00:46:52.381980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.381498    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"
Nov 03 00:46:52.381980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.381539    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf" event={"ID":"f98f73ad-617e-4322-b81f-60bef48a13be","Type":"ContainerDied","Data":"88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76"}
Nov 03 00:46:52.381980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.381599    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf" event={"ID":"f98f73ad-617e-4322-b81f-60bef48a13be","Type":"ContainerDied","Data":"46c031685a458691534e19ca9b505bb6b54bc16196071a5a64b202e2c3b97080"}
Nov 03 00:46:52.381980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.381634    6000 scope.go:122] "RemoveContainer" containerID="88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76"
Nov 03 00:46:52.393739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.393719    6000 scope.go:122] "RemoveContainer" containerID="88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76"
Nov 03 00:46:52.394028 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:46:52.394006    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76\": container with ID starting with 88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76 not found: ID does not exist" containerID="88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76"
Nov 03 00:46:52.394099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.394041    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76"} err="failed to get container status \"88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76\": rpc error: code = NotFound desc = could not find container \"88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76\": container with ID starting with 88834667d2e5147cda1cec9fc4609fabd7b2e5479e47664108a4eb7935babb76 not found: ID does not exist"
Nov 03 00:46:52.430807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.430782    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"]
Nov 03 00:46:52.442542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.442508    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548drpvnf"]
Nov 03 00:46:52.569663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:46:52.569628    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f98f73ad-617e-4322-b81f-60bef48a13be" path="/var/lib/kubelet/pods/f98f73ad-617e-4322-b81f-60bef48a13be/volumes"
Nov 03 00:47:00.662680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.662633    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"]
Nov 03 00:47:00.663833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.663814    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f98f73ad-617e-4322-b81f-60bef48a13be" containerName="payload-processing"
Nov 03 00:47:00.663929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.663836    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f98f73ad-617e-4322-b81f-60bef48a13be" containerName="payload-processing"
Nov 03 00:47:00.663929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.663876    6000 cpu_manager.go:414] "containerMap: removing container" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy"
Nov 03 00:47:00.663929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.663884    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy"
Nov 03 00:47:00.664089 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.664078    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f98f73ad-617e-4322-b81f-60bef48a13be" containerName="payload-processing"
Nov 03 00:47:00.664126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.664098    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="68d5d410-6702-46d1-a442-6d712cd0705f" containerName="istio-proxy"
Nov 03 00:47:00.669526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.669506    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.684521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.684491    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"]
Nov 03 00:47:00.752078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.752043    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2vr\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-kube-api-access-bb2vr\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.752247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.752162    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.752367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.752352    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.752413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.752378    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.853241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.853205    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.853241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.853243    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.853543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.853371    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2vr\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-kube-api-access-bb2vr\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.853543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.853428    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.854066 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.854045    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.854196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.854171    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.855893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.855855    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.872389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.872365    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2vr\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-kube-api-access-bb2vr\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:00.988600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:00.988521    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:01.296702 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:47:01.296673    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f2d752c_e872_4ef4_833b_2792d654268b.slice/crio-3055fdfb14b3a96336261aecec1269290450c78ea2b297acb49dacb819e4f09b WatchSource:0}: Error finding container 3055fdfb14b3a96336261aecec1269290450c78ea2b297acb49dacb819e4f09b: Status 404 returned error can't find the container with id 3055fdfb14b3a96336261aecec1269290450c78ea2b297acb49dacb819e4f09b
Nov 03 00:47:01.297031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:01.297007    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"]
Nov 03 00:47:01.542111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:01.542002    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77" event={"ID":"5f2d752c-e872-4ef4-833b-2792d654268b","Type":"ContainerStarted","Data":"18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25"}
Nov 03 00:47:01.542111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:01.542063    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77" event={"ID":"5f2d752c-e872-4ef4-833b-2792d654268b","Type":"ContainerStarted","Data":"3055fdfb14b3a96336261aecec1269290450c78ea2b297acb49dacb819e4f09b"}
Nov 03 00:47:01.542111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:01.542083    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:01.580735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:01.580688    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77" podStartSLOduration=1.580673826 podStartE2EDuration="1.580673826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:01.57909339 +0000 UTC m=+2459.134445413" watchObservedRunningTime="2026-11-03 00:47:01.580673826 +0000 UTC m=+2459.136025886"
Nov 03 00:47:02.821455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:02.821425    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"]
Nov 03 00:47:02.821960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:02.821623    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" containerID="cri-o://6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782" gracePeriod=30
Nov 03 00:47:06.079513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:06.079476    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 00:47:06.079904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:06.079536    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 00:47:08.117522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.117491    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:47:08.299319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299277    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-certs\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.299508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299384    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-socket\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.299508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299429    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istiod-ca-cert\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.299508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299476    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-data\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.299508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299482    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-certs" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:47:08.299718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299551    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-envoy\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.299718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299641    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-token\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-token\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.299718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299686    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-kube-api-access-ftnmj\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-kube-api-access-ftnmj\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.299884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299716    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-socket" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:47:08.299884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299811    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-data" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:47:08.299884 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299828    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istiod-ca-cert" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:08.300060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.299981    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-credential-socket\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.300060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.300038    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-podinfo\") pod \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\" (UID: \"7b1dba44-5c51-4c59-9cae-e122f75cd62b\") "
Nov 03 00:47:08.300305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.300236    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-credential-socket" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:47:08.300847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.300822    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.300988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.300971    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.301080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.300993    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.301080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.301006    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.301080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.301017    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.302430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.302402    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-kube-api-access-ftnmj" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "kube-api-access-ftnmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:08.302851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.302825    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-podinfo" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:47:08.304363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.304341    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-envoy" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:47:08.304466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.304394    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-token" pod "7b1dba44-5c51-4c59-9cae-e122f75cd62b" (UID: "7b1dba44-5c51-4c59-9cae-e122f75cd62b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:08.413373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.408242    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.413373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.408310    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.413373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.408323    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.413373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.408337    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftnmj\" (UniqueName: \"kubernetes.io/projected/7b1dba44-5c51-4c59-9cae-e122f75cd62b-kube-api-access-ftnmj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:08.629637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.629595    6000 generic.go:356] "Generic (PLEG): container finished" podID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerID="6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782" exitCode=0
Nov 03 00:47:08.629828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.629671    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"
Nov 03 00:47:08.629828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.629696    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" event={"ID":"7b1dba44-5c51-4c59-9cae-e122f75cd62b","Type":"ContainerDied","Data":"6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782"}
Nov 03 00:47:08.629828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.629763    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw" event={"ID":"7b1dba44-5c51-4c59-9cae-e122f75cd62b","Type":"ContainerDied","Data":"4adb3e8fb2080dfa92ec5323f60df2a990373c081899f5b236426d4d4c6ed180"}
Nov 03 00:47:08.629828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.629778    6000 scope.go:122] "RemoveContainer" containerID="6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782"
Nov 03 00:47:08.641877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.641849    6000 scope.go:122] "RemoveContainer" containerID="1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9"
Nov 03 00:47:08.654894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.654864    6000 scope.go:122] "RemoveContainer" containerID="6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782"
Nov 03 00:47:08.655232 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:08.655205    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782\": container with ID starting with 6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782 not found: ID does not exist" containerID="6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782"
Nov 03 00:47:08.655363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.655241    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782"} err="failed to get container status \"6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782\": rpc error: code = NotFound desc = could not find container \"6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782\": container with ID starting with 6c6a0bb547d5ea7d956cb01723e8bccd0330c272d0e37f0e3d83ed694f6cc782 not found: ID does not exist"
Nov 03 00:47:08.655363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.655292    6000 scope.go:122] "RemoveContainer" containerID="1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9"
Nov 03 00:47:08.655583 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:08.655560    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9\": container with ID starting with 1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9 not found: ID does not exist" containerID="1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9"
Nov 03 00:47:08.655630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.655594    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9"} err="failed to get container status \"1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9\": rpc error: code = NotFound desc = could not find container \"1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9\": container with ID starting with 1ed4031dde828f61b8b5b835a30449420eb40844f4f33592b0e32f5ef33682a9 not found: ID does not exist"
Nov 03 00:47:08.680795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.680726    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"]
Nov 03 00:47:08.687113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:08.687087    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-359f71-openshift-default-7cbcf59687-wtzmw"]
Nov 03 00:47:10.568506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:10.568477    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" path="/var/lib/kubelet/pods/7b1dba44-5c51-4c59-9cae-e122f75cd62b/volumes"
Nov 03 00:47:11.196714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.196662    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"]
Nov 03 00:47:11.197510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.197494    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy"
Nov 03 00:47:11.197578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.197511    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy"
Nov 03 00:47:11.197578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.197524    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy"
Nov 03 00:47:11.197578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.197529    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy"
Nov 03 00:47:11.197678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.197660    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy"
Nov 03 00:47:11.197678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.197671    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1dba44-5c51-4c59-9cae-e122f75cd62b" containerName="istio-proxy"
Nov 03 00:47:11.202899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.202879    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.220043 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.220015    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"]
Nov 03 00:47:11.229557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.229532    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"kube-root-ca.crt\""
Nov 03 00:47:11.229691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.229610    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"openshift-service-ca.crt\""
Nov 03 00:47:11.229691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.229658    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-e2e--70281560-dockercfg-vsts5\""
Nov 03 00:47:11.229691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.229673    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-955dc3\""
Nov 03 00:47:11.229691 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.229532    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"routing-overlay\""
Nov 03 00:47:11.229911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.229735    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"agc-cred-8a91ac4c0e0e\""
Nov 03 00:47:11.290656 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.288940    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssprf\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-kube-api-access-ssprf\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.291173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.291149    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.291399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.291380    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.291639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.291623    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.393578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.393542    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.393758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.393586    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.393758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.393648    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.393758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.393687    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssprf\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-kube-api-access-ssprf\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.394500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.394470    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.394645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.394552    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.396181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.396155    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.413934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.413911    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssprf\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-kube-api-access-ssprf\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.517843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.517764    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:11.667663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.667609    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"]
Nov 03 00:47:11.676650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.676627    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.690292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.690189    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"agc-cred-39505044c9f0\""
Nov 03 00:47:11.691309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.691284    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"]
Nov 03 00:47:11.802465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.802381    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.802465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.802435    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.802687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.802543    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghzdd\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-kube-api-access-ghzdd\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.802687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.802586    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.824286 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:47:11.824233    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29175110_32db_45df_a28c_fc365e02e555.slice/crio-db301ca9da3cdd71ad1b3a94a99ea5e37e231a6b5176c7e848d9b0b7d7c45562 WatchSource:0}: Error finding container db301ca9da3cdd71ad1b3a94a99ea5e37e231a6b5176c7e848d9b0b7d7c45562: Status 404 returned error can't find the container with id db301ca9da3cdd71ad1b3a94a99ea5e37e231a6b5176c7e848d9b0b7d7c45562
Nov 03 00:47:11.826045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.826014    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"]
Nov 03 00:47:11.904535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.904504    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.904711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.904612    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghzdd\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-kube-api-access-ghzdd\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.904711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.904664    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.904954 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.904930    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.905423 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.905399    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.905725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.905697    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.908026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.908005    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.923346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.923238    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghzdd\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-kube-api-access-ghzdd\") pod \"payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:11.994419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:11.994395    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:12.302528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.302474    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"]
Nov 03 00:47:12.553660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.553634    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:12.622288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.622229    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"]
Nov 03 00:47:12.668703 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.668667    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7" event={"ID":"06aacb92-40ab-4f70-b0f8-0de7ee69600a","Type":"ContainerStarted","Data":"2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4"}
Nov 03 00:47:12.669192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.668731    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7" event={"ID":"06aacb92-40ab-4f70-b0f8-0de7ee69600a","Type":"ContainerStarted","Data":"f34d45b4254927f837ee475af1627845b57097b82072140470afae1f11ec4b00"}
Nov 03 00:47:12.669192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.668776    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:12.670608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.670578    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv" event={"ID":"29175110-32db-45df-a28c-fc365e02e555","Type":"ContainerStarted","Data":"746d4293f19e65915ad37887d684eb3bbed55ac446f46bf3f39a0b936a590037"}
Nov 03 00:47:12.670745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.670628    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv" event={"ID":"29175110-32db-45df-a28c-fc365e02e555","Type":"ContainerStarted","Data":"db301ca9da3cdd71ad1b3a94a99ea5e37e231a6b5176c7e848d9b0b7d7c45562"}
Nov 03 00:47:12.670745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.670667    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:12.709994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.709950    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7" podStartSLOduration=1.7099339040000001 podStartE2EDuration="1.709933904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:12.707415706 +0000 UTC m=+2470.262767730" watchObservedRunningTime="2026-11-03 00:47:12.709933904 +0000 UTC m=+2470.265285928"
Nov 03 00:47:12.746936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:12.746882    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv" podStartSLOduration=1.74685957 podStartE2EDuration="1.74685957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:12.744236363 +0000 UTC m=+2470.299588386" watchObservedRunningTime="2026-11-03 00:47:12.74685957 +0000 UTC m=+2470.302211595"
Nov 03 00:47:15.574707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:15.574671    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"]
Nov 03 00:47:16.764838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:16.764803    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"]
Nov 03 00:47:17.622700 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:17.622651    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57" podUID="0655e7a3-f738-4b56-80f9-1363f8b75081" containerName="payload-processing" containerID="cri-o://bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31" gracePeriod=25
Nov 03 00:47:17.907765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:17.907740    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:47:18.023180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.023153    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-provider-credentials\") pod \"0655e7a3-f738-4b56-80f9-1363f8b75081\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") "
Nov 03 00:47:18.023414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.023400    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-kube-api-access-gbgg4\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-kube-api-access-gbgg4\") pod \"0655e7a3-f738-4b56-80f9-1363f8b75081\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") "
Nov 03 00:47:18.023465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.023446    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-plugins-config-volume\") pod \"0655e7a3-f738-4b56-80f9-1363f8b75081\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") "
Nov 03 00:47:18.023514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.023478    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-routing-overlay-volume\") pod \"0655e7a3-f738-4b56-80f9-1363f8b75081\" (UID: \"0655e7a3-f738-4b56-80f9-1363f8b75081\") "
Nov 03 00:47:18.023891 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.023830    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-routing-overlay-volume" pod "0655e7a3-f738-4b56-80f9-1363f8b75081" (UID: "0655e7a3-f738-4b56-80f9-1363f8b75081"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:18.024009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.023983    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-plugins-config-volume" pod "0655e7a3-f738-4b56-80f9-1363f8b75081" (UID: "0655e7a3-f738-4b56-80f9-1363f8b75081"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:18.024155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.024136    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:18.024208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.024161    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0655e7a3-f738-4b56-80f9-1363f8b75081-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:18.025500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.025479    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-provider-credentials" pod "0655e7a3-f738-4b56-80f9-1363f8b75081" (UID: "0655e7a3-f738-4b56-80f9-1363f8b75081"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:18.025695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.025680    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-kube-api-access-gbgg4" pod "0655e7a3-f738-4b56-80f9-1363f8b75081" (UID: "0655e7a3-f738-4b56-80f9-1363f8b75081"). InnerVolumeSpecName "kube-api-access-gbgg4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:18.125977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.125901    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbgg4\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-kube-api-access-gbgg4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:18.125977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.125932    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0655e7a3-f738-4b56-80f9-1363f8b75081-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:18.716140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.716106    6000 generic.go:356] "Generic (PLEG): container finished" podID="0655e7a3-f738-4b56-80f9-1363f8b75081" containerID="bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31" exitCode=0
Nov 03 00:47:18.716383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.716186    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57" event={"ID":"0655e7a3-f738-4b56-80f9-1363f8b75081","Type":"ContainerDied","Data":"bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31"}
Nov 03 00:47:18.716383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.716191    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"
Nov 03 00:47:18.716383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.716231    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57" event={"ID":"0655e7a3-f738-4b56-80f9-1363f8b75081","Type":"ContainerDied","Data":"a77a8caf538fced2850e58a36eb2c2a9f1dac27e03ec026db2ebb8e7c5cf9bbc"}
Nov 03 00:47:18.716383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.716285    6000 scope.go:122] "RemoveContainer" containerID="bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31"
Nov 03 00:47:18.726989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.726971    6000 scope.go:122] "RemoveContainer" containerID="bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31"
Nov 03 00:47:18.727289 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:18.727232    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31\": container with ID starting with bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31 not found: ID does not exist" containerID="bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31"
Nov 03 00:47:18.727384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.727299    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31"} err="failed to get container status \"bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31\": rpc error: code = NotFound desc = could not find container \"bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31\": container with ID starting with bf2c65fe67d1ea0b981de50e885c0928e5190b476b56e085ae32963ee261ec31 not found: ID does not exist"
Nov 03 00:47:18.767388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.767353    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"]
Nov 03 00:47:18.778813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:18.778790    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-d999hjf57"]
Nov 03 00:47:19.896850 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:19.896814    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:47:20.570132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.570097    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0655e7a3-f738-4b56-80f9-1363f8b75081" path="/var/lib/kubelet/pods/0655e7a3-f738-4b56-80f9-1363f8b75081/volumes"
Nov 03 00:47:20.574943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.574916    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv" podUID="29175110-32db-45df-a28c-fc365e02e555" containerName="payload-processing" containerID="cri-o://746d4293f19e65915ad37887d684eb3bbed55ac446f46bf3f39a0b936a590037" gracePeriod=25
Nov 03 00:47:20.734489 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.734453    6000 generic.go:356] "Generic (PLEG): container finished" podID="29175110-32db-45df-a28c-fc365e02e555" containerID="746d4293f19e65915ad37887d684eb3bbed55ac446f46bf3f39a0b936a590037" exitCode=0
Nov 03 00:47:20.734655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.734516    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv" event={"ID":"29175110-32db-45df-a28c-fc365e02e555","Type":"ContainerDied","Data":"746d4293f19e65915ad37887d684eb3bbed55ac446f46bf3f39a0b936a590037"}
Nov 03 00:47:20.849882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.849858    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:20.866216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.866191    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-kube-api-access-ssprf\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-kube-api-access-ssprf\") pod \"29175110-32db-45df-a28c-fc365e02e555\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") "
Nov 03 00:47:20.866397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.866284    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-provider-credentials\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-provider-credentials\") pod \"29175110-32db-45df-a28c-fc365e02e555\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") "
Nov 03 00:47:20.866469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.866454    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-routing-overlay-volume\") pod \"29175110-32db-45df-a28c-fc365e02e555\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") "
Nov 03 00:47:20.866517 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.866493    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-plugins-config-volume\") pod \"29175110-32db-45df-a28c-fc365e02e555\" (UID: \"29175110-32db-45df-a28c-fc365e02e555\") "
Nov 03 00:47:20.866858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.866832    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-routing-overlay-volume" pod "29175110-32db-45df-a28c-fc365e02e555" (UID: "29175110-32db-45df-a28c-fc365e02e555"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:20.867185 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.867160    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-plugins-config-volume" pod "29175110-32db-45df-a28c-fc365e02e555" (UID: "29175110-32db-45df-a28c-fc365e02e555"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:20.867333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.867218    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:20.868766 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.868739    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-provider-credentials" pod "29175110-32db-45df-a28c-fc365e02e555" (UID: "29175110-32db-45df-a28c-fc365e02e555"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:20.869002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.868983    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-kube-api-access-ssprf" pod "29175110-32db-45df-a28c-fc365e02e555" (UID: "29175110-32db-45df-a28c-fc365e02e555"). InnerVolumeSpecName "kube-api-access-ssprf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:20.969063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.969034    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29175110-32db-45df-a28c-fc365e02e555-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:20.969063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.969061    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ssprf\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-kube-api-access-ssprf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:20.969063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:20.969070    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/29175110-32db-45df-a28c-fc365e02e555-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:21.743490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:21.743461    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"
Nov 03 00:47:21.743680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:21.743493    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv" event={"ID":"29175110-32db-45df-a28c-fc365e02e555","Type":"ContainerDied","Data":"db301ca9da3cdd71ad1b3a94a99ea5e37e231a6b5176c7e848d9b0b7d7c45562"}
Nov 03 00:47:21.743680 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:21.743554    6000 scope.go:122] "RemoveContainer" containerID="746d4293f19e65915ad37887d684eb3bbed55ac446f46bf3f39a0b936a590037"
Nov 03 00:47:21.765402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:21.765372    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7" podUID="06aacb92-40ab-4f70-b0f8-0de7ee69600a" containerName="payload-processing" containerID="cri-o://2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4" gracePeriod=25
Nov 03 00:47:21.795881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:21.795857    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"]
Nov 03 00:47:21.807020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:21.806997    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-567bwj7sv"]
Nov 03 00:47:22.041971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.041945    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:22.085676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.085644    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-routing-overlay-volume\") pod \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") "
Nov 03 00:47:22.085837 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.085824    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-provider-credentials\") pod \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") "
Nov 03 00:47:22.085904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.085860    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-kube-api-access-ghzdd\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-kube-api-access-ghzdd\") pod \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") "
Nov 03 00:47:22.085998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.085972    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-plugins-config-volume\") pod \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\" (UID: \"06aacb92-40ab-4f70-b0f8-0de7ee69600a\") "
Nov 03 00:47:22.086239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.086217    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-routing-overlay-volume" pod "06aacb92-40ab-4f70-b0f8-0de7ee69600a" (UID: "06aacb92-40ab-4f70-b0f8-0de7ee69600a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:22.086676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.086653    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:22.087196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.087171    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-plugins-config-volume" pod "06aacb92-40ab-4f70-b0f8-0de7ee69600a" (UID: "06aacb92-40ab-4f70-b0f8-0de7ee69600a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:22.088653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.088623    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-provider-credentials" pod "06aacb92-40ab-4f70-b0f8-0de7ee69600a" (UID: "06aacb92-40ab-4f70-b0f8-0de7ee69600a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:22.088751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.088673    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-kube-api-access-ghzdd" pod "06aacb92-40ab-4f70-b0f8-0de7ee69600a" (UID: "06aacb92-40ab-4f70-b0f8-0de7ee69600a"). InnerVolumeSpecName "kube-api-access-ghzdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:22.189222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.189188    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:22.189222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.189223    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghzdd\" (UniqueName: \"kubernetes.io/projected/06aacb92-40ab-4f70-b0f8-0de7ee69600a-kube-api-access-ghzdd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:22.190813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.189236    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06aacb92-40ab-4f70-b0f8-0de7ee69600a-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:22.572149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.572117    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29175110-32db-45df-a28c-fc365e02e555" path="/var/lib/kubelet/pods/29175110-32db-45df-a28c-fc365e02e555/volumes"
Nov 03 00:47:22.751890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.751856    6000 generic.go:356] "Generic (PLEG): container finished" podID="06aacb92-40ab-4f70-b0f8-0de7ee69600a" containerID="2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4" exitCode=0
Nov 03 00:47:22.752072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.751899    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"
Nov 03 00:47:22.752072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.751934    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7" event={"ID":"06aacb92-40ab-4f70-b0f8-0de7ee69600a","Type":"ContainerDied","Data":"2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4"}
Nov 03 00:47:22.752072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.751993    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7" event={"ID":"06aacb92-40ab-4f70-b0f8-0de7ee69600a","Type":"ContainerDied","Data":"f34d45b4254927f837ee475af1627845b57097b82072140470afae1f11ec4b00"}
Nov 03 00:47:22.752072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.752024    6000 scope.go:122] "RemoveContainer" containerID="2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4"
Nov 03 00:47:22.762693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.762675    6000 scope.go:122] "RemoveContainer" containerID="2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4"
Nov 03 00:47:22.763034 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:22.763012    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4\": container with ID starting with 2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4 not found: ID does not exist" containerID="2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4"
Nov 03 00:47:22.763112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.763043    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4"} err="failed to get container status \"2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4\": rpc error: code = NotFound desc = could not find container \"2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4\": container with ID starting with 2e76a85f87d63117b0ffd40654ff70124ac7c726252f785c20cb7895b1ccf1d4 not found: ID does not exist"
Nov 03 00:47:22.801241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.801210    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"]
Nov 03 00:47:22.807012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:22.806985    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5c55w6qb7"]
Nov 03 00:47:24.569470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:24.569386    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06aacb92-40ab-4f70-b0f8-0de7ee69600a" path="/var/lib/kubelet/pods/06aacb92-40ab-4f70-b0f8-0de7ee69600a/volumes"
Nov 03 00:47:26.697434 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.697376    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"]
Nov 03 00:47:26.698200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698183    6000 cpu_manager.go:414] "containerMap: removing container" podUID="06aacb92-40ab-4f70-b0f8-0de7ee69600a" containerName="payload-processing"
Nov 03 00:47:26.698283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698201    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06aacb92-40ab-4f70-b0f8-0de7ee69600a" containerName="payload-processing"
Nov 03 00:47:26.698283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698241    6000 cpu_manager.go:414] "containerMap: removing container" podUID="29175110-32db-45df-a28c-fc365e02e555" containerName="payload-processing"
Nov 03 00:47:26.698283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698246    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29175110-32db-45df-a28c-fc365e02e555" containerName="payload-processing"
Nov 03 00:47:26.698283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698283    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0655e7a3-f738-4b56-80f9-1363f8b75081" containerName="payload-processing"
Nov 03 00:47:26.698420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698289    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0655e7a3-f738-4b56-80f9-1363f8b75081" containerName="payload-processing"
Nov 03 00:47:26.698420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698390    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0655e7a3-f738-4b56-80f9-1363f8b75081" containerName="payload-processing"
Nov 03 00:47:26.698420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698402    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="06aacb92-40ab-4f70-b0f8-0de7ee69600a" containerName="payload-processing"
Nov 03 00:47:26.698420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.698415    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="29175110-32db-45df-a28c-fc365e02e555" containerName="payload-processing"
Nov 03 00:47:26.702389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.702367    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.721488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.721465    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"]
Nov 03 00:47:26.757888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.757847    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.758067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.757977    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs6hn\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-kube-api-access-cs6hn\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.758067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.758020    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.758210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.758086    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.859337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.859288    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.859513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.859425    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.859513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.859501    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs6hn\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-kube-api-access-cs6hn\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.859619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.859543    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.860131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.860102    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.860269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.860180    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.862151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.862128    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:26.881705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:26.881681    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs6hn\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-kube-api-access-cs6hn\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:27.018074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:27.017988    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:27.319482 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:47:27.319448    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91990cc8_20e2_496e_ab9a_975460c89774.slice/crio-ce10bf1826640c6b4bf30b1ca5c9a135b32b08f76a0b7ff7df033acd69008900 WatchSource:0}: Error finding container ce10bf1826640c6b4bf30b1ca5c9a135b32b08f76a0b7ff7df033acd69008900: Status 404 returned error can't find the container with id ce10bf1826640c6b4bf30b1ca5c9a135b32b08f76a0b7ff7df033acd69008900
Nov 03 00:47:27.319646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:27.319513    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"]
Nov 03 00:47:27.791885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:27.791843    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj" event={"ID":"91990cc8-20e2-496e-ab9a-975460c89774","Type":"ContainerStarted","Data":"cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30"}
Nov 03 00:47:27.792362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:27.791900    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj" event={"ID":"91990cc8-20e2-496e-ab9a-975460c89774","Type":"ContainerStarted","Data":"ce10bf1826640c6b4bf30b1ca5c9a135b32b08f76a0b7ff7df033acd69008900"}
Nov 03 00:47:27.792362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:27.791938    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:27.830150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:27.830092    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj" podStartSLOduration=1.8300771789999999 podStartE2EDuration="1.830077179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:27.828720178 +0000 UTC m=+2485.384072202" watchObservedRunningTime="2026-11-03 00:47:27.830077179 +0000 UTC m=+2485.385429193"
Nov 03 00:47:28.884830 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.884800    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"]
Nov 03 00:47:28.889585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.889562    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.903473 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.903445    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-ac338d-openshift-default-dockercfg-hs9cf\""
Nov 03 00:47:28.919890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.919862    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"]
Nov 03 00:47:28.990335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990302    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg889\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-kube-api-access-dg889\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990341    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990490    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990547    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990600    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990751    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990790    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990837    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356bea3c-a365-4f7a-81f5-35c9bd90b427-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:28.990949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:28.990894    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.092900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.092853    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg889\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-kube-api-access-dg889\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.092900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.092908    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.092981    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093007    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093059    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093139    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093160    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093317    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356bea3c-a365-4f7a-81f5-35c9bd90b427-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093366    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093453    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-data\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093624    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093656    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-credential-socket\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.093870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093841    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-certs\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.094022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.093998    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356bea3c-a365-4f7a-81f5-35c9bd90b427-istiod-ca-cert\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.095764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.095741    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-envoy\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.095975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.095953    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-podinfo\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.112240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.112211    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-token\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.112435 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.112413    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg889\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-kube-api-access-dg889\") pod \"e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") " pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.206300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.206174    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:29.529921 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:47:29.529889    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356bea3c_a365_4f7a_81f5_35c9bd90b427.slice/crio-2d61eb2d656da825a36fe519699fe320ad8e65a37ac407e03d1442025b0263a4 WatchSource:0}: Error finding container 2d61eb2d656da825a36fe519699fe320ad8e65a37ac407e03d1442025b0263a4: Status 404 returned error can't find the container with id 2d61eb2d656da825a36fe519699fe320ad8e65a37ac407e03d1442025b0263a4
Nov 03 00:47:29.530040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.529911    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"]
Nov 03 00:47:29.532941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.532909    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:47:29.533036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.532981    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:47:29.533036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.533011    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:47:29.809733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.809634    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" event={"ID":"356bea3c-a365-4f7a-81f5-35c9bd90b427","Type":"ContainerStarted","Data":"ab91dd6abce2634b2b283838312f4a316177327ae531ae4ce3fbd67f7da84486"}
Nov 03 00:47:29.809896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.809741    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" event={"ID":"356bea3c-a365-4f7a-81f5-35c9bd90b427","Type":"ContainerStarted","Data":"2d61eb2d656da825a36fe519699fe320ad8e65a37ac407e03d1442025b0263a4"}
Nov 03 00:47:29.855580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:29.855517    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podStartSLOduration=1.855497022 podStartE2EDuration="1.855497022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:29.853207837 +0000 UTC m=+2487.408559856" watchObservedRunningTime="2026-11-03 00:47:29.855497022 +0000 UTC m=+2487.410849046"
Nov 03 00:47:30.131521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.131481    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"]
Nov 03 00:47:30.136345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.136321    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.155688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.155660    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"]
Nov 03 00:47:30.206832 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.206775    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:30.208128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.208101    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:47:30.208299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.208155    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 00:47:30.217426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.217395    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.217634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.217615    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxgn6\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-kube-api-access-lxgn6\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.217756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.217733    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.217820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.217786    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.318841 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.318808    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxgn6\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-kube-api-access-lxgn6\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.319075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.318906    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.319075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.318949    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.319075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.319042    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.319742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.319716    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.319852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.319756    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.321623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.321604    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.337521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.337491    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxgn6\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-kube-api-access-lxgn6\") pod \"payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.457179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.452195    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:30.817670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:30.817638    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"]
Nov 03 00:47:30.819034 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:47:30.818994    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f31f2e_fd90_4a18_bfd6_8fc3df8d6f16.slice/crio-609b36e7c04530ce5eda606b4cfb183b7e2776c300e7c8ce2fce4c48f9491d24 WatchSource:0}: Error finding container 609b36e7c04530ce5eda606b4cfb183b7e2776c300e7c8ce2fce4c48f9491d24: Status 404 returned error can't find the container with id 609b36e7c04530ce5eda606b4cfb183b7e2776c300e7c8ce2fce4c48f9491d24
Nov 03 00:47:31.209317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:31.208856    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:47:31.209317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:31.208914    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 00:47:31.831330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:31.831295    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt" event={"ID":"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16","Type":"ContainerStarted","Data":"517eebcbf0f18f7844e439751f08e06c2b81eaec5f196370d40b037361970c60"}
Nov 03 00:47:31.831606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:31.831351    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt" event={"ID":"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16","Type":"ContainerStarted","Data":"609b36e7c04530ce5eda606b4cfb183b7e2776c300e7c8ce2fce4c48f9491d24"}
Nov 03 00:47:31.869381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:31.869327    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt" podStartSLOduration=1.86930831 podStartE2EDuration="1.86930831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:31.867986272 +0000 UTC m=+2489.423338297" watchObservedRunningTime="2026-11-03 00:47:31.86930831 +0000 UTC m=+2489.424660341"
Nov 03 00:47:32.212666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:32.212623    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:47:32.213196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:32.212678    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 00:47:32.838712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:32.838677    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:33.210292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:33.210143    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:47:33.210292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:33.210205    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 00:47:34.211712 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:34.211667    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:47:34.212220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:34.211730    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 00:47:35.108278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:35.106581    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:47:35.208288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:35.207624    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body=
Nov 03 00:47:35.208288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:35.207688    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused"
Nov 03 00:47:36.330191 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:36.330158    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:36.330984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:36.330953    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:36.332319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:36.332299    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 00:47:38.803077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:38.803048    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:47:38.878282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:38.878232    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"]
Nov 03 00:47:41.741809 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:41.741772    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"]
Nov 03 00:47:41.752376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:41.752348    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"]
Nov 03 00:47:41.756227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:41.756201    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:42.640493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:42.640463    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"]
Nov 03 00:47:43.878916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:43.878875    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql" podUID="ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" containerName="payload-processing" containerID="cri-o://8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be" gracePeriod=25
Nov 03 00:47:44.181037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.181010    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:47:44.305201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.305170    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-plugins-config-volume\") pod \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") "
Nov 03 00:47:44.305436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.305375    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-provider-credentials\") pod \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") "
Nov 03 00:47:44.305436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.305425    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-kube-api-access-v7l22\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-kube-api-access-v7l22\") pod \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") "
Nov 03 00:47:44.305581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.305526    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-routing-overlay-volume\") pod \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\" (UID: \"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae\") "
Nov 03 00:47:44.305683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.305658    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-plugins-config-volume" pod "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" (UID: "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:44.305901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.305877    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-routing-overlay-volume" pod "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" (UID: "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:44.306107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.306090    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:44.306180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.306111    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:44.307839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.307815    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-kube-api-access-v7l22" pod "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" (UID: "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae"). InnerVolumeSpecName "kube-api-access-v7l22". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:44.307944 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.307874    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-provider-credentials" pod "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" (UID: "ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:44.407484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.407406    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:44.407484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.407436    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v7l22\" (UniqueName: \"kubernetes.io/projected/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae-kube-api-access-v7l22\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:44.937647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.937601    6000 generic.go:356] "Generic (PLEG): container finished" podID="ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" containerID="8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be" exitCode=0
Nov 03 00:47:44.937647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.937651    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"
Nov 03 00:47:44.938226 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.937669    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql" event={"ID":"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae","Type":"ContainerDied","Data":"8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be"}
Nov 03 00:47:44.938226 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.937728    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql" event={"ID":"ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae","Type":"ContainerDied","Data":"a33aa79688a40de926a63f738569542dee1c1be6f5f287fe1de7db9c3cf23bbe"}
Nov 03 00:47:44.938226 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.937750    6000 scope.go:122] "RemoveContainer" containerID="8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be"
Nov 03 00:47:44.950889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.950858    6000 scope.go:122] "RemoveContainer" containerID="8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be"
Nov 03 00:47:44.951165 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:44.951147    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be\": container with ID starting with 8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be not found: ID does not exist" containerID="8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be"
Nov 03 00:47:44.951231 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.951172    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be"} err="failed to get container status \"8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be\": rpc error: code = NotFound desc = could not find container \"8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be\": container with ID starting with 8069d94167f5628488a86fd4ed235238dcddf83456c2d799a2c023842d0854be not found: ID does not exist"
Nov 03 00:47:44.987023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.986999    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"]
Nov 03 00:47:44.998220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:44.998199    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-594cdc4ql"]
Nov 03 00:47:46.571009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:46.570976    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" path="/var/lib/kubelet/pods/ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae/volumes"
Nov 03 00:47:46.742832 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:46.742790    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl" podUID="dc1f9283-5f2a-4b95-a41d-2622a69c8095" containerName="payload-processing" containerID="cri-o://26dc4c458108c09a0252cec6e582764e8f67485dc60bdd4a5a9e30aaea3e7147" gracePeriod=25
Nov 03 00:47:46.752999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:46.752961    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt" podUID="72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" containerName="payload-processing" containerID="cri-o://517eebcbf0f18f7844e439751f08e06c2b81eaec5f196370d40b037361970c60" gracePeriod=25
Nov 03 00:47:46.968029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:46.967677    6000 generic.go:356] "Generic (PLEG): container finished" podID="72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" containerID="517eebcbf0f18f7844e439751f08e06c2b81eaec5f196370d40b037361970c60" exitCode=0
Nov 03 00:47:46.968029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:46.967928    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt" event={"ID":"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16","Type":"ContainerDied","Data":"517eebcbf0f18f7844e439751f08e06c2b81eaec5f196370d40b037361970c60"}
Nov 03 00:47:46.970399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:46.970219    6000 generic.go:356] "Generic (PLEG): container finished" podID="dc1f9283-5f2a-4b95-a41d-2622a69c8095" containerID="26dc4c458108c09a0252cec6e582764e8f67485dc60bdd4a5a9e30aaea3e7147" exitCode=0
Nov 03 00:47:46.970399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:46.970330    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl" event={"ID":"dc1f9283-5f2a-4b95-a41d-2622a69c8095","Type":"ContainerDied","Data":"26dc4c458108c09a0252cec6e582764e8f67485dc60bdd4a5a9e30aaea3e7147"}
Nov 03 00:47:47.079671 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.079604    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:47:47.109797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.109772    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:47.145073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145039    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-routing-overlay-volume\") pod \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") "
Nov 03 00:47:47.145246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145100    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-routing-overlay-volume\") pod \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") "
Nov 03 00:47:47.145246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145199    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-kube-api-access-lxgn6\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-kube-api-access-lxgn6\") pod \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") "
Nov 03 00:47:47.145395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145249    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-provider-credentials\") pod \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") "
Nov 03 00:47:47.145395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145296    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-provider-credentials\") pod \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") "
Nov 03 00:47:47.145395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145341    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-kube-api-access-6xqjx\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-kube-api-access-6xqjx\") pod \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") "
Nov 03 00:47:47.145551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145397    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-plugins-config-volume\") pod \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\" (UID: \"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16\") "
Nov 03 00:47:47.145551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145515    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-plugins-config-volume\") pod \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\" (UID: \"dc1f9283-5f2a-4b95-a41d-2622a69c8095\") "
Nov 03 00:47:47.145654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145544    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-routing-overlay-volume" pod "dc1f9283-5f2a-4b95-a41d-2622a69c8095" (UID: "dc1f9283-5f2a-4b95-a41d-2622a69c8095"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:47.145654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.145566    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-routing-overlay-volume" pod "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" (UID: "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:47.146040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.146016    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-plugins-config-volume" pod "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" (UID: "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:47.146189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.146070    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-plugins-config-volume" pod "dc1f9283-5f2a-4b95-a41d-2622a69c8095" (UID: "dc1f9283-5f2a-4b95-a41d-2622a69c8095"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:47.146338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.146161    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.146464 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.146449    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.148120 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.148095    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-kube-api-access-6xqjx" pod "dc1f9283-5f2a-4b95-a41d-2622a69c8095" (UID: "dc1f9283-5f2a-4b95-a41d-2622a69c8095"). InnerVolumeSpecName "kube-api-access-6xqjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:47.148288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.148129    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-provider-credentials" pod "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" (UID: "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:47.148288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.148123    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-kube-api-access-lxgn6" pod "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" (UID: "72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16"). InnerVolumeSpecName "kube-api-access-lxgn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:47.148426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.148308    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-provider-credentials" pod "dc1f9283-5f2a-4b95-a41d-2622a69c8095" (UID: "dc1f9283-5f2a-4b95-a41d-2622a69c8095"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:47.247858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.247830    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxgn6\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-kube-api-access-lxgn6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.247858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.247856    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.247858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.247866    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.248108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.247876    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xqjx\" (UniqueName: \"kubernetes.io/projected/dc1f9283-5f2a-4b95-a41d-2622a69c8095-kube-api-access-6xqjx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.248108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.247886    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.248108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.247895    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dc1f9283-5f2a-4b95-a41d-2622a69c8095-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:47.641390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.641350    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77" podUID="5f2d752c-e872-4ef4-833b-2792d654268b" containerName="payload-processing" containerID="cri-o://18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25" gracePeriod=25
Nov 03 00:47:47.941877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.941849    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:47.979748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.979711    6000 generic.go:356] "Generic (PLEG): container finished" podID="5f2d752c-e872-4ef4-833b-2792d654268b" containerID="18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25" exitCode=0
Nov 03 00:47:47.979943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.979799    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77" event={"ID":"5f2d752c-e872-4ef4-833b-2792d654268b","Type":"ContainerDied","Data":"18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25"}
Nov 03 00:47:47.979943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.979895    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77" event={"ID":"5f2d752c-e872-4ef4-833b-2792d654268b","Type":"ContainerDied","Data":"3055fdfb14b3a96336261aecec1269290450c78ea2b297acb49dacb819e4f09b"}
Nov 03 00:47:47.979943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.979925    6000 scope.go:122] "RemoveContainer" containerID="18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25"
Nov 03 00:47:47.979943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.979809    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"
Nov 03 00:47:47.981838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.981807    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"
Nov 03 00:47:47.982003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.981809    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl" event={"ID":"dc1f9283-5f2a-4b95-a41d-2622a69c8095","Type":"ContainerDied","Data":"3155b8b2487d582f1b62e2b247a672b4733ca442a228fab1b40560ce23d0347c"}
Nov 03 00:47:47.983645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.983621    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt" event={"ID":"72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16","Type":"ContainerDied","Data":"609b36e7c04530ce5eda606b4cfb183b7e2776c300e7c8ce2fce4c48f9491d24"}
Nov 03 00:47:47.983806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.983785    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"
Nov 03 00:47:47.993797 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.993778    6000 scope.go:122] "RemoveContainer" containerID="18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25"
Nov 03 00:47:47.994333 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:47.994140    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25\": container with ID starting with 18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25 not found: ID does not exist" containerID="18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25"
Nov 03 00:47:47.994333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.994180    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25"} err="failed to get container status \"18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25\": rpc error: code = NotFound desc = could not find container \"18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25\": container with ID starting with 18b2b82cc74bf3695c7d4e1a957d78cabda2ae566156175377d5c22f47e0cb25 not found: ID does not exist"
Nov 03 00:47:47.994333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:47.994202    6000 scope.go:122] "RemoveContainer" containerID="26dc4c458108c09a0252cec6e582764e8f67485dc60bdd4a5a9e30aaea3e7147"
Nov 03 00:47:48.009500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.008055    6000 scope.go:122] "RemoveContainer" containerID="517eebcbf0f18f7844e439751f08e06c2b81eaec5f196370d40b037361970c60"
Nov 03 00:47:48.047457 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.046539    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"]
Nov 03 00:47:48.049840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.049812    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-5f4dzwdpl"]
Nov 03 00:47:48.058789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.058764    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-plugins-config-volume\") pod \"5f2d752c-e872-4ef4-833b-2792d654268b\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") "
Nov 03 00:47:48.058907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.058838    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-provider-credentials\") pod \"5f2d752c-e872-4ef4-833b-2792d654268b\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") "
Nov 03 00:47:48.059027 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.059009    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-routing-overlay-volume\") pod \"5f2d752c-e872-4ef4-833b-2792d654268b\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") "
Nov 03 00:47:48.059165 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.059146    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-kube-api-access-bb2vr\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-kube-api-access-bb2vr\") pod \"5f2d752c-e872-4ef4-833b-2792d654268b\" (UID: \"5f2d752c-e872-4ef4-833b-2792d654268b\") "
Nov 03 00:47:48.059325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.059145    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-plugins-config-volume" pod "5f2d752c-e872-4ef4-833b-2792d654268b" (UID: "5f2d752c-e872-4ef4-833b-2792d654268b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:48.059465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.059400    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-routing-overlay-volume" pod "5f2d752c-e872-4ef4-833b-2792d654268b" (UID: "5f2d752c-e872-4ef4-833b-2792d654268b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:47:48.059976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.059953    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:48.060088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.059983    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5f2d752c-e872-4ef4-833b-2792d654268b-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:48.061170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.061149    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-provider-credentials" pod "5f2d752c-e872-4ef4-833b-2792d654268b" (UID: "5f2d752c-e872-4ef4-833b-2792d654268b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:48.061353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.061332    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-kube-api-access-bb2vr" pod "5f2d752c-e872-4ef4-833b-2792d654268b" (UID: "5f2d752c-e872-4ef4-833b-2792d654268b"). InnerVolumeSpecName "kube-api-access-bb2vr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:47:48.081312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.081287    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"]
Nov 03 00:47:48.084468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.084443    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-548dqtrwt"]
Nov 03 00:47:48.161612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.161541    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2vr\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-kube-api-access-bb2vr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:48.161612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.161565    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/5f2d752c-e872-4ef4-833b-2792d654268b-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:47:48.327443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.327398    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"]
Nov 03 00:47:48.334140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.334096    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-5bfdvhw77"]
Nov 03 00:47:48.569356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.569269    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f2d752c-e872-4ef4-833b-2792d654268b" path="/var/lib/kubelet/pods/5f2d752c-e872-4ef4-833b-2792d654268b/volumes"
Nov 03 00:47:48.569826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.569803    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" path="/var/lib/kubelet/pods/72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16/volumes"
Nov 03 00:47:48.570298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:48.570269    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc1f9283-5f2a-4b95-a41d-2622a69c8095" path="/var/lib/kubelet/pods/dc1f9283-5f2a-4b95-a41d-2622a69c8095/volumes"
Nov 03 00:47:49.897993 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:47:49.897955    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:47:55.656596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:55.656556    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"]
Nov 03 00:47:58.246299 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.245212    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"]
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246478    6000 cpu_manager.go:414] "containerMap: removing container" podUID="72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246500    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246537    6000 cpu_manager.go:414] "containerMap: removing container" podUID="5f2d752c-e872-4ef4-833b-2792d654268b" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246545    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f2d752c-e872-4ef4-833b-2792d654268b" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246570    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246578    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246621    6000 cpu_manager.go:414] "containerMap: removing container" podUID="dc1f9283-5f2a-4b95-a41d-2622a69c8095" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246629    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc1f9283-5f2a-4b95-a41d-2622a69c8095" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246779    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f2d752c-e872-4ef4-833b-2792d654268b" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246794    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed6ad117-5d8f-4554-bbcb-a77dd74fc9ae" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246810    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc1f9283-5f2a-4b95-a41d-2622a69c8095" containerName="payload-processing"
Nov 03 00:47:58.246822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.246823    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f31f2e-fd90-4a18-bfd6-8fc3df8d6f16" containerName="payload-processing"
Nov 03 00:47:58.257447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.257415    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"]
Nov 03 00:47:58.257629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.257534    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.289591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.289555    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"routing-overlay\""
Nov 03 00:47:58.289783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.289652    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"openshift-service-ca.crt\""
Nov 03 00:47:58.289783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.289701    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"agc-cred-0219878e6cb3\""
Nov 03 00:47:58.289929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.289786    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-e2e--70281560-dockercfg-lrv8c\""
Nov 03 00:47:58.289929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.289853    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-955dc3\""
Nov 03 00:47:58.289929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.289902    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"kube-root-ca.crt\""
Nov 03 00:47:58.371984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.371947    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5jtk\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-kube-api-access-m5jtk\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.372175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.372017    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.372175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.372150    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.372443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.372423    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.474210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.474165    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.474460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.474228    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5jtk\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-kube-api-access-m5jtk\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.474460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.474300    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.474460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.474346    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.475098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.475069    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.475154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.475141    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.477092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.477067    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.493828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.493801    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5jtk\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-kube-api-access-m5jtk\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.602348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.597244    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:58.928699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:58.928650    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"]
Nov 03 00:47:59.114525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:59.114481    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp" event={"ID":"40e5f52c-9d0b-44dd-a4e5-cffb398588f2","Type":"ContainerStarted","Data":"834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6"}
Nov 03 00:47:59.114750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:59.114547    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp" event={"ID":"40e5f52c-9d0b-44dd-a4e5-cffb398588f2","Type":"ContainerStarted","Data":"64179c2f575e62e591dd64c1f89a531ba21f7a2d89c2814193cce8cbffd11413"}
Nov 03 00:47:59.114750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:59.114574    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:47:59.151819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:47:59.151770    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp" podStartSLOduration=1.1517562670000001 podStartE2EDuration="1.151756267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:47:59.150659459 +0000 UTC m=+2516.706011483" watchObservedRunningTime="2026-11-03 00:47:59.151756267 +0000 UTC m=+2516.707108292"
Nov 03 00:48:00.008061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.008027    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"]
Nov 03 00:48:00.013050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.013025    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.027608 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.027583    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"kube-root-ca.crt\""
Nov 03 00:48:00.033036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.033004    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"]
Nov 03 00:48:00.033323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.033297    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"openshift-service-ca.crt\""
Nov 03 00:48:00.033323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.033310    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d9e067\""
Nov 03 00:48:00.033513 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.033372    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"routing-overlay\""
Nov 03 00:48:00.066506 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.066475    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"agc-cred-063dd6fb9af3\""
Nov 03 00:48:00.066665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.066576    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"payload-processing-external-model-e2e--b567d708-dockercfg-4fgbm\""
Nov 03 00:48:00.098552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.098511    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n726v\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-kube-api-access-n726v\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.098729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.098582    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.098789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.098748    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.098834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.098815    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.208295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.191797    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"]
Nov 03 00:48:00.208794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.206181    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.209045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.209026    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.209447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.209424    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n726v\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-kube-api-access-n726v\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.209615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.209600    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.210702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.210673    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.221400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.220643    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.269885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.258938    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.283146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.283007    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n726v\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-kube-api-access-n726v\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.332068 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.332037    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:00.655578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.655543    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"]
Nov 03 00:48:00.656949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.656918    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj" podUID="91990cc8-20e2-496e-ab9a-975460c89774" containerName="payload-processing" containerID="cri-o://cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30" gracePeriod=25
Nov 03 00:48:00.950032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.950006    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:48:00.999113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.999080    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-routing-overlay-volume\") pod \"91990cc8-20e2-496e-ab9a-975460c89774\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") "
Nov 03 00:48:00.999328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.999129    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-kube-api-access-cs6hn\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-kube-api-access-cs6hn\") pod \"91990cc8-20e2-496e-ab9a-975460c89774\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") "
Nov 03 00:48:00.999328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.999181    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-plugins-config-volume\") pod \"91990cc8-20e2-496e-ab9a-975460c89774\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") "
Nov 03 00:48:00.999467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.999405    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-provider-credentials\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-provider-credentials\") pod \"91990cc8-20e2-496e-ab9a-975460c89774\" (UID: \"91990cc8-20e2-496e-ab9a-975460c89774\") "
Nov 03 00:48:00.999581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.999550    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-routing-overlay-volume" pod "91990cc8-20e2-496e-ab9a-975460c89774" (UID: "91990cc8-20e2-496e-ab9a-975460c89774"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:00.999740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:00.999710    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-plugins-config-volume" pod "91990cc8-20e2-496e-ab9a-975460c89774" (UID: "91990cc8-20e2-496e-ab9a-975460c89774"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:01.000155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.000131    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:01.000228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.000160    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/91990cc8-20e2-496e-ab9a-975460c89774-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:01.002508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.002482    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-kube-api-access-cs6hn" pod "91990cc8-20e2-496e-ab9a-975460c89774" (UID: "91990cc8-20e2-496e-ab9a-975460c89774"). InnerVolumeSpecName "kube-api-access-cs6hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:01.003734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.003705    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-provider-credentials" pod "91990cc8-20e2-496e-ab9a-975460c89774" (UID: "91990cc8-20e2-496e-ab9a-975460c89774"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:01.101543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.101507    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:01.101543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.101535    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs6hn\" (UniqueName: \"kubernetes.io/projected/91990cc8-20e2-496e-ab9a-975460c89774-kube-api-access-cs6hn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:01.134007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.133969    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw" event={"ID":"6ae45849-4c5d-40d0-bb5f-f78a448adbcb","Type":"ContainerStarted","Data":"0edd3ce0f0918e682adaa405187e7fdb972c6f68cab8a19a6a6a798a02e08106"}
Nov 03 00:48:01.134182 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.134035    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw" event={"ID":"6ae45849-4c5d-40d0-bb5f-f78a448adbcb","Type":"ContainerStarted","Data":"c789b30d5cba2f8d2bb5e641666baa054cad626164ee883e123efc16c8160756"}
Nov 03 00:48:01.134280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.134189    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:01.135482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.135457    6000 generic.go:356] "Generic (PLEG): container finished" podID="91990cc8-20e2-496e-ab9a-975460c89774" containerID="cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30" exitCode=0
Nov 03 00:48:01.135619 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.135572    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"
Nov 03 00:48:01.135679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.135623    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj" event={"ID":"91990cc8-20e2-496e-ab9a-975460c89774","Type":"ContainerDied","Data":"cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30"}
Nov 03 00:48:01.135727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.135679    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj" event={"ID":"91990cc8-20e2-496e-ab9a-975460c89774","Type":"ContainerDied","Data":"ce10bf1826640c6b4bf30b1ca5c9a135b32b08f76a0b7ff7df033acd69008900"}
Nov 03 00:48:01.135727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.135709    6000 scope.go:122] "RemoveContainer" containerID="cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30"
Nov 03 00:48:01.147380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.147358    6000 scope.go:122] "RemoveContainer" containerID="cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30"
Nov 03 00:48:01.147675 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:01.147655    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30\": container with ID starting with cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30 not found: ID does not exist" containerID="cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30"
Nov 03 00:48:01.147764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.147685    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30"} err="failed to get container status \"cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30\": rpc error: code = NotFound desc = could not find container \"cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30\": container with ID starting with cc45580f576f1072c788528730afdd1e6b488b7f91e8a84984b60a558e84ea30 not found: ID does not exist"
Nov 03 00:48:01.179931 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.179830    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw" podStartSLOduration=2.179814797 podStartE2EDuration="2.179814797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:01.175138089 +0000 UTC m=+2518.730490110" watchObservedRunningTime="2026-11-03 00:48:01.179814797 +0000 UTC m=+2518.735166821"
Nov 03 00:48:01.216428 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.216393    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"]
Nov 03 00:48:01.223867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:01.223838    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-65ccss2kj"]
Nov 03 00:48:02.570481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:02.570450    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91990cc8-20e2-496e-ab9a-975460c89774" path="/var/lib/kubelet/pods/91990cc8-20e2-496e-ab9a-975460c89774/volumes"
Nov 03 00:48:03.725339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.725303    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"]
Nov 03 00:48:03.726103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.726089    6000 cpu_manager.go:414] "containerMap: removing container" podUID="91990cc8-20e2-496e-ab9a-975460c89774" containerName="payload-processing"
Nov 03 00:48:03.726154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.726104    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91990cc8-20e2-496e-ab9a-975460c89774" containerName="payload-processing"
Nov 03 00:48:03.726234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.726225    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="91990cc8-20e2-496e-ab9a-975460c89774" containerName="payload-processing"
Nov 03 00:48:03.731971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.731953    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.750179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.750147    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"]
Nov 03 00:48:03.762932 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.762899    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t445r\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-kube-api-access-t445r\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.763114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.763015    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.763245 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.763223    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.763393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.763374    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.787650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.787617    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"agc-cred-1c38736881be\""
Nov 03 00:48:03.865360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.865313    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.865550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.865432    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.865550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.865518    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t445r\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-kube-api-access-t445r\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.865550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.865546    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.866234 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.866206    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.866419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.866340    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.868313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.868293    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:03.887625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:03.887592    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t445r\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-kube-api-access-t445r\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:04.049393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.049241    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:04.354709 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:48:04.354680    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbad7667_36ff_4064_9293_6d5499f28812.slice/crio-62604d377a35fa0f66ac300899dac8110e1f676c812a4fe685c80c13b0206cce WatchSource:0}: Error finding container 62604d377a35fa0f66ac300899dac8110e1f676c812a4fe685c80c13b0206cce: Status 404 returned error can't find the container with id 62604d377a35fa0f66ac300899dac8110e1f676c812a4fe685c80c13b0206cce
Nov 03 00:48:04.355341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.355315    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"]
Nov 03 00:48:04.847733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.847693    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"]
Nov 03 00:48:04.852754 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.852733    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:04.881246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.881222    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"e2e-cfl-b3e08e-provider-cred\""
Nov 03 00:48:04.881376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.881227    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"payload-processing-external-model-e2e--bf641009-dockercfg-hdkvs\""
Nov 03 00:48:04.896012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.895988    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"]
Nov 03 00:48:04.988655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.988613    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8247p\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-kube-api-access-8247p\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:04.989114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.989092    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:04.989345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.989326    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:04.994280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:04.989576    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.091328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.091297    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8247p\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-kube-api-access-8247p\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.091328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.091337    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.091563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.091394    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.091563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.091443    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.092044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.092015    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.092180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.092109    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.094232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.094212    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.110798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.110777    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8247p\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-kube-api-access-8247p\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.168665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.168619    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:05.189029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.188994    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml" event={"ID":"fbad7667-36ff-4064-9293-6d5499f28812","Type":"ContainerStarted","Data":"8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e"}
Nov 03 00:48:05.189178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.189058    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml" event={"ID":"fbad7667-36ff-4064-9293-6d5499f28812","Type":"ContainerStarted","Data":"62604d377a35fa0f66ac300899dac8110e1f676c812a4fe685c80c13b0206cce"}
Nov 03 00:48:05.190185 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.190163    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:05.209423 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.209383    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t" podUID="c04d2687-9948-4422-8eb7-57ed70de723c" containerName="payload-processing" containerID="cri-o://cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55" gracePeriod=25
Nov 03 00:48:05.235713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.235647    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml" podStartSLOduration=2.235625174 podStartE2EDuration="2.235625174s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:05.226207976 +0000 UTC m=+2522.781560035" watchObservedRunningTime="2026-11-03 00:48:05.235625174 +0000 UTC m=+2522.790977201"
Nov 03 00:48:05.463035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.463004    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:48:05.479844 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:48:05.479815    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd8bca4_d096_4296_971e_015d9e4b8017.slice/crio-6e1dc21f0729a72b567da1fe3bb1367c9278edf409f663bbb94ea310cb7e66e8 WatchSource:0}: Error finding container 6e1dc21f0729a72b567da1fe3bb1367c9278edf409f663bbb94ea310cb7e66e8: Status 404 returned error can't find the container with id 6e1dc21f0729a72b567da1fe3bb1367c9278edf409f663bbb94ea310cb7e66e8
Nov 03 00:48:05.480617 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.480593    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"]
Nov 03 00:48:05.546232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.545716    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-plugins-config-volume\") pod \"c04d2687-9948-4422-8eb7-57ed70de723c\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") "
Nov 03 00:48:05.546232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.545947    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-provider-credentials\") pod \"c04d2687-9948-4422-8eb7-57ed70de723c\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") "
Nov 03 00:48:05.546232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.546037    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-routing-overlay-volume\") pod \"c04d2687-9948-4422-8eb7-57ed70de723c\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") "
Nov 03 00:48:05.546232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.546068    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-kube-api-access-bc54v\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-kube-api-access-bc54v\") pod \"c04d2687-9948-4422-8eb7-57ed70de723c\" (UID: \"c04d2687-9948-4422-8eb7-57ed70de723c\") "
Nov 03 00:48:05.548859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.547012    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-routing-overlay-volume" pod "c04d2687-9948-4422-8eb7-57ed70de723c" (UID: "c04d2687-9948-4422-8eb7-57ed70de723c"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:05.548859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.548018    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-plugins-config-volume" pod "c04d2687-9948-4422-8eb7-57ed70de723c" (UID: "c04d2687-9948-4422-8eb7-57ed70de723c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:05.555316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.549197    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-provider-credentials" pod "c04d2687-9948-4422-8eb7-57ed70de723c" (UID: "c04d2687-9948-4422-8eb7-57ed70de723c"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:05.556468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.556300    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-kube-api-access-bc54v" pod "c04d2687-9948-4422-8eb7-57ed70de723c" (UID: "c04d2687-9948-4422-8eb7-57ed70de723c"). InnerVolumeSpecName "kube-api-access-bc54v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:05.648693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.648668    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:05.648801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.648696    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:05.648801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.648710    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c04d2687-9948-4422-8eb7-57ed70de723c-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:05.648801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:05.648722    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc54v\" (UniqueName: \"kubernetes.io/projected/c04d2687-9948-4422-8eb7-57ed70de723c-kube-api-access-bc54v\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:06.198344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.198304    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg" event={"ID":"abd8bca4-d096-4296-971e-015d9e4b8017","Type":"ContainerStarted","Data":"ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69"}
Nov 03 00:48:06.198956 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.198374    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg" event={"ID":"abd8bca4-d096-4296-971e-015d9e4b8017","Type":"ContainerStarted","Data":"6e1dc21f0729a72b567da1fe3bb1367c9278edf409f663bbb94ea310cb7e66e8"}
Nov 03 00:48:06.198956 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.198403    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:06.199929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.199896    6000 generic.go:356] "Generic (PLEG): container finished" podID="c04d2687-9948-4422-8eb7-57ed70de723c" containerID="cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55" exitCode=0
Nov 03 00:48:06.200074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.199942    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"
Nov 03 00:48:06.200074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.199971    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t" event={"ID":"c04d2687-9948-4422-8eb7-57ed70de723c","Type":"ContainerDied","Data":"cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55"}
Nov 03 00:48:06.200074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.200016    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t" event={"ID":"c04d2687-9948-4422-8eb7-57ed70de723c","Type":"ContainerDied","Data":"f888e2c3de1c2a2d32021783863cd30abfbf7d03ebd7f46bd72a074c2142f5f7"}
Nov 03 00:48:06.200074 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.200039    6000 scope.go:122] "RemoveContainer" containerID="cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55"
Nov 03 00:48:06.211055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.211037    6000 scope.go:122] "RemoveContainer" containerID="cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55"
Nov 03 00:48:06.211358 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:06.211338    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55\": container with ID starting with cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55 not found: ID does not exist" containerID="cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55"
Nov 03 00:48:06.211405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.211368    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55"} err="failed to get container status \"cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55\": rpc error: code = NotFound desc = could not find container \"cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55\": container with ID starting with cf4e5e019464facff702454ee31e155640710185e578e204400a451238ef0c55 not found: ID does not exist"
Nov 03 00:48:06.235663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.235611    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg" podStartSLOduration=2.235597846 podStartE2EDuration="2.235597846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:06.234446452 +0000 UTC m=+2523.789798477" watchObservedRunningTime="2026-11-03 00:48:06.235597846 +0000 UTC m=+2523.790949869"
Nov 03 00:48:06.270286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.270226    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"]
Nov 03 00:48:06.273516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.273489    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-b9cd2vp4t"]
Nov 03 00:48:06.464113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.464035    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"]
Nov 03 00:48:06.464862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.464847    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c04d2687-9948-4422-8eb7-57ed70de723c" containerName="payload-processing"
Nov 03 00:48:06.464921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.464865    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04d2687-9948-4422-8eb7-57ed70de723c" containerName="payload-processing"
Nov 03 00:48:06.465050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.465038    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04d2687-9948-4422-8eb7-57ed70de723c" containerName="payload-processing"
Nov 03 00:48:06.470702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.470681    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.484527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.484481    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"routing-overlay\""
Nov 03 00:48:06.484527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.484481    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"payload-processing-external-model-plugins-e2e-extmod-a-850433\""
Nov 03 00:48:06.484527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.484492    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"kube-root-ca.crt\""
Nov 03 00:48:06.484753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.484581    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"openshift-service-ca.crt\""
Nov 03 00:48:06.485995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.485974    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"]
Nov 03 00:48:06.589173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.586342    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.589173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.586498    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.589173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.586538    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.589173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.586686    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q6vj\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-kube-api-access-8q6vj\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.589173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.588699    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"agc-cred-7f3c8bb4f9e8\""
Nov 03 00:48:06.589626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.589326    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"payload-processing-external-model-e2e--977b8100-dockercfg-gkbbw\""
Nov 03 00:48:06.663056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.662354    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04d2687-9948-4422-8eb7-57ed70de723c" path="/var/lib/kubelet/pods/c04d2687-9948-4422-8eb7-57ed70de723c/volumes"
Nov 03 00:48:06.688618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.688591    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q6vj\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-kube-api-access-8q6vj\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.688794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.688736    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.688836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.688808    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.688877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.688835    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.689420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.689396    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.689595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.689568    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.691563 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.691547    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.706993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.706953    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q6vj\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-kube-api-access-8q6vj\") pod \"payload-processing-external-model-e2e-extmod-a-850433-6d75tz669\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:06.786625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:06.786532    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:07.011382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.011348    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"]
Nov 03 00:48:07.017609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.017585    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.031319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.031293    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"agc-cred-45cfe41774e1\""
Nov 03 00:48:07.034307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.034286    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"]
Nov 03 00:48:07.090802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.090774    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"]
Nov 03 00:48:07.090952 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:48:07.090897    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8ce2b3a_cbae_4057_858f_e10a904a686e.slice/crio-e55de891be8803623ed8982a6758d852a1586ec733bdba6350a65babf3d46ccd WatchSource:0}: Error finding container e55de891be8803623ed8982a6758d852a1586ec733bdba6350a65babf3d46ccd: Status 404 returned error can't find the container with id e55de891be8803623ed8982a6758d852a1586ec733bdba6350a65babf3d46ccd
Nov 03 00:48:07.198274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.197176    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.198274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.197459    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k5gw\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-kube-api-access-5k5gw\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.198274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.197520    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.198274 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.197545    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.222399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.222366    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" event={"ID":"d8ce2b3a-cbae-4057-858f-e10a904a686e","Type":"ContainerStarted","Data":"e55de891be8803623ed8982a6758d852a1586ec733bdba6350a65babf3d46ccd"}
Nov 03 00:48:07.299344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.299300    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.299344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.299353    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.299609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.299474    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.299730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.299698    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k5gw\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-kube-api-access-5k5gw\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.300141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.300104    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.300526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.300509    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.302207 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.302188    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.324039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.324016    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k5gw\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-kube-api-access-5k5gw\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.334521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.334501    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:07.641267 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:07.641221    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"]
Nov 03 00:48:08.232613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.232576    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb" event={"ID":"c9998f17-d123-47f3-b44d-4b4bc2a6ae57","Type":"ContainerStarted","Data":"2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a"}
Nov 03 00:48:08.233130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.232645    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb" event={"ID":"c9998f17-d123-47f3-b44d-4b4bc2a6ae57","Type":"ContainerStarted","Data":"61cbea395a8a687456dc3d4345345ce952b16cbc8ddf13e779b1c61c6e90f6d8"}
Nov 03 00:48:08.233130 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.232678    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:08.234538 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.234512    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" event={"ID":"d8ce2b3a-cbae-4057-858f-e10a904a686e","Type":"ContainerStarted","Data":"e3726612d91b28c34d3209e4c06a8ed957b59b9c5c2e872b662d9d384bc5478f"}
Nov 03 00:48:08.234664 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.234578    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:08.272184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.272156    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"]
Nov 03 00:48:08.280471 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.280427    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb" podStartSLOduration=2.28041375 podStartE2EDuration="2.28041375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:08.270352819 +0000 UTC m=+2525.825704842" watchObservedRunningTime="2026-11-03 00:48:08.28041375 +0000 UTC m=+2525.835765774"
Nov 03 00:48:08.309883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:08.309824    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" podStartSLOduration=2.30980386 podStartE2EDuration="2.30980386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:08.307032794 +0000 UTC m=+2525.862384820" watchObservedRunningTime="2026-11-03 00:48:08.30980386 +0000 UTC m=+2525.865155885"
Nov 03 00:48:08.321844 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.321813    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-850433/agc-cred-7f3c8bb4f9e8: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:08.321844 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.321832    6000 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-plugins-e2e-extmod-a-850433: configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:08.321844 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.321844    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:08.322133 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.321906    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:08.821886876 +0000 UTC m=+2526.377238891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:08.322492 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.322456    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:08.822439894 +0000 UTC m=+2526.377791922 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:08.831166 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.831128    6000 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-plugins-e2e-extmod-a-850433: configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:08.831389 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.831185    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-850433/agc-cred-7f3c8bb4f9e8: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:08.831389 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.831207    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:08.831389 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.831237    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:09.831193211 +0000 UTC m=+2527.386545228 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:08.831389 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:08.831279    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:09.831267082 +0000 UTC m=+2527.386619087 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:09.243045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:09.243013    6000 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" secret="" err="secret \"payload-processing-external-model-e2e--977b8100-dockercfg-gkbbw\" not found"
Nov 03 00:48:09.846026 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:09.845993    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-850433/agc-cred-7f3c8bb4f9e8: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:09.846026 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:09.846017    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:09.846026 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:09.846020    6000 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-plugins-e2e-extmod-a-850433: configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:09.846247 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:09.846071    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:11.846055712 +0000 UTC m=+2529.401407732 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:09.846247 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:09.846084    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:11.84607837 +0000 UTC m=+2529.401430372 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:10.128330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:10.128303    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:48:11.154154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.154122    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"]
Nov 03 00:48:11.159201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.159182    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.175032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.175003    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"e2e-cmp-legacy-2c9719-provider-cred\""
Nov 03 00:48:11.176420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.176400    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"]
Nov 03 00:48:11.183366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.183329    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d87781\""
Nov 03 00:48:11.183366 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.183329    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"payload-processing-external-model-e2e--2422cafc-dockercfg-zlbrh\""
Nov 03 00:48:11.183518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.183459    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"openshift-service-ca.crt\""
Nov 03 00:48:11.183783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.183762    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"routing-overlay\""
Nov 03 00:48:11.183873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.183780    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"kube-root-ca.crt\""
Nov 03 00:48:11.271281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.257323    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9z8h\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-kube-api-access-p9z8h\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.271281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.257400    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.271281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.257486    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.271281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.257552    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.365281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.364654    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.365281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.364810    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.365281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.364916    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.365281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.365234    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9z8h\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-kube-api-access-p9z8h\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.383282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.366587    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.383282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.377467    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.399342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.397181    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.399342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.399205    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9z8h\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-kube-api-access-p9z8h\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.475593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.475477    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:11.779170 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:48:11.779139    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc46b83a_34ec_4f92_a8fb_0ec63ec09a0d.slice/crio-f06d1f25c8c40ae34cda73a9d99d609fc64131447c77651539bf9475285f3cd6 WatchSource:0}: Error finding container f06d1f25c8c40ae34cda73a9d99d609fc64131447c77651539bf9475285f3cd6: Status 404 returned error can't find the container with id f06d1f25c8c40ae34cda73a9d99d609fc64131447c77651539bf9475285f3cd6
Nov 03 00:48:11.779340 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:11.779162    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"]
Nov 03 00:48:11.875373 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:11.875342    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-850433/agc-cred-7f3c8bb4f9e8: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:11.875373 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:11.875374    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669: secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:11.875559 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:11.875342    6000 configmap.go:193] Couldn't get configMap ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-plugins-e2e-extmod-a-850433: configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:11.875559 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:11.875423    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:15.87540973 +0000 UTC m=+2533.430761755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : secret "agc-cred-7f3c8bb4f9e8" not found
Nov 03 00:48:11.875559 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:11.875442    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume podName:d8ce2b3a-cbae-4057-858f-e10a904a686e nodeName:}" failed. No retries permitted until 2026-11-03 00:48:15.875433136 +0000 UTC m=+2533.430785140 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume") pod "payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e") : configmap "payload-processing-external-model-plugins-e2e-extmod-a-850433" not found
Nov 03 00:48:12.162510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:12.162483    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:12.358063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:12.358027    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c" event={"ID":"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d","Type":"ContainerStarted","Data":"d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e"}
Nov 03 00:48:12.358244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:12.358084    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c" event={"ID":"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d","Type":"ContainerStarted","Data":"f06d1f25c8c40ae34cda73a9d99d609fc64131447c77651539bf9475285f3cd6"}
Nov 03 00:48:12.358244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:12.358236    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:12.396403 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:12.396336    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c" podStartSLOduration=1.396314525 podStartE2EDuration="1.396314525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:12.393948026 +0000 UTC m=+2529.949300051" watchObservedRunningTime="2026-11-03 00:48:12.396314525 +0000 UTC m=+2529.951666553"
Nov 03 00:48:15.258099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.258048    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" podUID="d8ce2b3a-cbae-4057-858f-e10a904a686e" containerName="payload-processing" containerID="cri-o://e3726612d91b28c34d3209e4c06a8ed957b59b9c5c2e872b662d9d384bc5478f" gracePeriod=25
Nov 03 00:48:15.413937 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.413897    6000 generic.go:356] "Generic (PLEG): container finished" podID="d8ce2b3a-cbae-4057-858f-e10a904a686e" containerID="e3726612d91b28c34d3209e4c06a8ed957b59b9c5c2e872b662d9d384bc5478f" exitCode=0
Nov 03 00:48:15.414145 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.414126    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" event={"ID":"d8ce2b3a-cbae-4057-858f-e10a904a686e","Type":"ContainerDied","Data":"e3726612d91b28c34d3209e4c06a8ed957b59b9c5c2e872b662d9d384bc5478f"}
Nov 03 00:48:15.565895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.565869    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:15.633192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.633154    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials\") pod \"d8ce2b3a-cbae-4057-858f-e10a904a686e\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") "
Nov 03 00:48:15.633414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.633392    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-routing-overlay-volume\") pod \"d8ce2b3a-cbae-4057-858f-e10a904a686e\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") "
Nov 03 00:48:15.633497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.633447    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-kube-api-access-8q6vj\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-kube-api-access-8q6vj\") pod \"d8ce2b3a-cbae-4057-858f-e10a904a686e\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") "
Nov 03 00:48:15.633559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.633530    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume\") pod \"d8ce2b3a-cbae-4057-858f-e10a904a686e\" (UID: \"d8ce2b3a-cbae-4057-858f-e10a904a686e\") "
Nov 03 00:48:15.633814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.633787    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-routing-overlay-volume" pod "d8ce2b3a-cbae-4057-858f-e10a904a686e" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:15.634108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.634073    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume" pod "d8ce2b3a-cbae-4057-858f-e10a904a686e" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:15.634269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.634182    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:15.635973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.635945    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials" pod "d8ce2b3a-cbae-4057-858f-e10a904a686e" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:15.636204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.636183    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-kube-api-access-8q6vj" pod "d8ce2b3a-cbae-4057-858f-e10a904a686e" (UID: "d8ce2b3a-cbae-4057-858f-e10a904a686e"). InnerVolumeSpecName "kube-api-access-8q6vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:15.735591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.735565    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q6vj\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-kube-api-access-8q6vj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:15.735591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.735589    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d8ce2b3a-cbae-4057-858f-e10a904a686e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:15.735591 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:15.735599    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/d8ce2b3a-cbae-4057-858f-e10a904a686e-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:16.424512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:16.424468    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"
Nov 03 00:48:16.425030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:16.424513    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669" event={"ID":"d8ce2b3a-cbae-4057-858f-e10a904a686e","Type":"ContainerDied","Data":"e55de891be8803623ed8982a6758d852a1586ec733bdba6350a65babf3d46ccd"}
Nov 03 00:48:16.425030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:16.424629    6000 scope.go:122] "RemoveContainer" containerID="e3726612d91b28c34d3209e4c06a8ed957b59b9c5c2e872b662d9d384bc5478f"
Nov 03 00:48:16.475356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:16.475319    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"]
Nov 03 00:48:16.486170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:16.486145    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-6d75tz669"]
Nov 03 00:48:16.569380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:16.569349    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ce2b3a-cbae-4057-858f-e10a904a686e" path="/var/lib/kubelet/pods/d8ce2b3a-cbae-4057-858f-e10a904a686e/volumes"
Nov 03 00:48:17.225270 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:17.225234    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:48:17.225476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:17.225324    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:48:17.296402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:17.296372    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"]
Nov 03 00:48:19.246735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.246710    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:48:19.309627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.309586    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"]
Nov 03 00:48:19.897149 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:19.897112    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:48:19.959049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.959009    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"]
Nov 03 00:48:19.960126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.960103    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d8ce2b3a-cbae-4057-858f-e10a904a686e" containerName="payload-processing"
Nov 03 00:48:19.960126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.960123    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ce2b3a-cbae-4057-858f-e10a904a686e" containerName="payload-processing"
Nov 03 00:48:19.960398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.960277    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ce2b3a-cbae-4057-858f-e10a904a686e" containerName="payload-processing"
Nov 03 00:48:19.965425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.965408    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.987611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.987587    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"]
Nov 03 00:48:19.992468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992438    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992582 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992482    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992610    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992634    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992706 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992664    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992757    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992787    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k458x\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-kube-api-access-k458x\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992911    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:19.992967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:19.992940    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094062    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094096 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094101    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k458x\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-kube-api-access-k458x\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094193    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094215    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094333    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094367    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094429    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094446    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094478    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094774    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-certs\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094869    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.094961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.094945    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-data\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.095062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.095046    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-credential-socket\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.095194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.095174    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istiod-ca-cert\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.097266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.097226    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-envoy\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.097396 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.097381    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-podinfo\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.113012 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.112987    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-token\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.113109 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.113018    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k458x\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-kube-api-access-k458x\") pod \"e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") " pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.282200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.282096    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:20.602150 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:48:20.602105    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb302f98_0db5_4b53_8aaf_00d315f1a60f.slice/crio-9704e0e7b97d6107f5dec4a4b2f35b5840ea6ae1917dd937e5416363ac51b084 WatchSource:0}: Error finding container 9704e0e7b97d6107f5dec4a4b2f35b5840ea6ae1917dd937e5416363ac51b084: Status 404 returned error can't find the container with id 9704e0e7b97d6107f5dec4a4b2f35b5840ea6ae1917dd937e5416363ac51b084
Nov 03 00:48:20.603724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.603700    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"]
Nov 03 00:48:20.605930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.605892    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:20.606009 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.605973    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:20.606048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:20.606016    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:21.490542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:21.490499    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" event={"ID":"bb302f98-0db5-4b53-8aaf-00d315f1a60f","Type":"ContainerStarted","Data":"9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd"}
Nov 03 00:48:21.491075 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:21.490563    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" event={"ID":"bb302f98-0db5-4b53-8aaf-00d315f1a60f","Type":"ContainerStarted","Data":"9704e0e7b97d6107f5dec4a4b2f35b5840ea6ae1917dd937e5416363ac51b084"}
Nov 03 00:48:21.557286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:21.557203    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" podStartSLOduration=2.557188746 podStartE2EDuration="2.557188746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:21.541858353 +0000 UTC m=+2539.097210376" watchObservedRunningTime="2026-11-03 00:48:21.557188746 +0000 UTC m=+2539.112540769"
Nov 03 00:48:22.282572 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.282536    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:22.284126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.284104    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 00:48:22.284201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.284153    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 00:48:22.311282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.310317    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw" podUID="6ae45849-4c5d-40d0-bb5f-f78a448adbcb" containerName="payload-processing" containerID="cri-o://0edd3ce0f0918e682adaa405187e7fdb972c6f68cab8a19a6a6a798a02e08106" gracePeriod=25
Nov 03 00:48:22.528129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.528094    6000 generic.go:356] "Generic (PLEG): container finished" podID="6ae45849-4c5d-40d0-bb5f-f78a448adbcb" containerID="0edd3ce0f0918e682adaa405187e7fdb972c6f68cab8a19a6a6a798a02e08106" exitCode=0
Nov 03 00:48:22.528775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.528164    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw" event={"ID":"6ae45849-4c5d-40d0-bb5f-f78a448adbcb","Type":"ContainerDied","Data":"0edd3ce0f0918e682adaa405187e7fdb972c6f68cab8a19a6a6a798a02e08106"}
Nov 03 00:48:22.654311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.654286    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:22.739079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.739042    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-plugins-config-volume\") pod \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") "
Nov 03 00:48:22.745280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.739499    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-routing-overlay-volume\") pod \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") "
Nov 03 00:48:22.759290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.740271    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-plugins-config-volume" pod "6ae45849-4c5d-40d0-bb5f-f78a448adbcb" (UID: "6ae45849-4c5d-40d0-bb5f-f78a448adbcb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:22.759290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.745436    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-routing-overlay-volume" pod "6ae45849-4c5d-40d0-bb5f-f78a448adbcb" (UID: "6ae45849-4c5d-40d0-bb5f-f78a448adbcb"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:22.759290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.751527    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-provider-credentials\") pod \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") "
Nov 03 00:48:22.759290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.751648    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-kube-api-access-n726v\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-kube-api-access-n726v\") pod \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\" (UID: \"6ae45849-4c5d-40d0-bb5f-f78a448adbcb\") "
Nov 03 00:48:22.759290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.752601    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:22.759290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.752624    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:22.759759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.759510    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-kube-api-access-n726v" pod "6ae45849-4c5d-40d0-bb5f-f78a448adbcb" (UID: "6ae45849-4c5d-40d0-bb5f-f78a448adbcb"). InnerVolumeSpecName "kube-api-access-n726v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:22.759759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.759671    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-provider-credentials" pod "6ae45849-4c5d-40d0-bb5f-f78a448adbcb" (UID: "6ae45849-4c5d-40d0-bb5f-f78a448adbcb"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:22.854057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.854020    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:22.854331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:22.854314    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n726v\" (UniqueName: \"kubernetes.io/projected/6ae45849-4c5d-40d0-bb5f-f78a448adbcb-kube-api-access-n726v\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:23.283114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.283076    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 00:48:23.283375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.283143    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 00:48:23.368746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.368716    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:48:23.539114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.539020    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw" event={"ID":"6ae45849-4c5d-40d0-bb5f-f78a448adbcb","Type":"ContainerDied","Data":"c789b30d5cba2f8d2bb5e641666baa054cad626164ee883e123efc16c8160756"}
Nov 03 00:48:23.539114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.539095    6000 scope.go:122] "RemoveContainer" containerID="0edd3ce0f0918e682adaa405187e7fdb972c6f68cab8a19a6a6a798a02e08106"
Nov 03 00:48:23.539719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.538986    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"
Nov 03 00:48:23.614902 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.614837    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"]
Nov 03 00:48:23.619214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:23.619182    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974q65sw"]
Nov 03 00:48:24.283580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.283442    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 00:48:24.283580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.283497    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 00:48:24.311409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.311349    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4" podUID="8be5dc52-85ed-4c07-8b3b-8549fc818b9e" containerName="payload-processing" containerID="cri-o://c48be655b4f0fadd689b7f9d0c5305de512a3a92fa2c327fd135807a4058d12f" gracePeriod=25
Nov 03 00:48:24.618577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.618470    6000 generic.go:356] "Generic (PLEG): container finished" podID="8be5dc52-85ed-4c07-8b3b-8549fc818b9e" containerID="c48be655b4f0fadd689b7f9d0c5305de512a3a92fa2c327fd135807a4058d12f" exitCode=0
Nov 03 00:48:24.621611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.621581    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae45849-4c5d-40d0-bb5f-f78a448adbcb" path="/var/lib/kubelet/pods/6ae45849-4c5d-40d0-bb5f-f78a448adbcb/volumes"
Nov 03 00:48:24.621992 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.621972    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4" event={"ID":"8be5dc52-85ed-4c07-8b3b-8549fc818b9e","Type":"ContainerDied","Data":"c48be655b4f0fadd689b7f9d0c5305de512a3a92fa2c327fd135807a4058d12f"}
Nov 03 00:48:24.677880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.677854    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:48:24.684346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.684322    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-plugins-config-volume\") pod \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") "
Nov 03 00:48:24.684522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.684404    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-provider-credentials\") pod \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") "
Nov 03 00:48:24.684723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.684709    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-routing-overlay-volume\") pod \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") "
Nov 03 00:48:24.684773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.684746    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-plugins-config-volume" pod "8be5dc52-85ed-4c07-8b3b-8549fc818b9e" (UID: "8be5dc52-85ed-4c07-8b3b-8549fc818b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:24.684859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.684841    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-kube-api-access-7nlv2\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-kube-api-access-7nlv2\") pod \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\" (UID: \"8be5dc52-85ed-4c07-8b3b-8549fc818b9e\") "
Nov 03 00:48:24.685088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.685067    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-routing-overlay-volume" pod "8be5dc52-85ed-4c07-8b3b-8549fc818b9e" (UID: "8be5dc52-85ed-4c07-8b3b-8549fc818b9e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:48:24.685484 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.685468    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:24.685585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.685575    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:24.712560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.712523    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-provider-credentials" pod "8be5dc52-85ed-4c07-8b3b-8549fc818b9e" (UID: "8be5dc52-85ed-4c07-8b3b-8549fc818b9e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:24.712839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.712822    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-kube-api-access-7nlv2" pod "8be5dc52-85ed-4c07-8b3b-8549fc818b9e" (UID: "8be5dc52-85ed-4c07-8b3b-8549fc818b9e"). InnerVolumeSpecName "kube-api-access-7nlv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:48:24.811385 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.787182    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nlv2\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-kube-api-access-7nlv2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:24.811385 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:24.811340    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8be5dc52-85ed-4c07-8b3b-8549fc818b9e-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:48:25.283087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:25.283050    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 00:48:25.283302 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:25.283111    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 00:48:25.630273 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:25.630226    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4" event={"ID":"8be5dc52-85ed-4c07-8b3b-8549fc818b9e","Type":"ContainerDied","Data":"6926587d1502350f57839cd5feb89473dd3c8d770159747e5ab9394ce04420af"}
Nov 03 00:48:25.630755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:25.630303    6000 scope.go:122] "RemoveContainer" containerID="c48be655b4f0fadd689b7f9d0c5305de512a3a92fa2c327fd135807a4058d12f"
Nov 03 00:48:25.630755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:25.630420    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"
Nov 03 00:48:25.681261 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:25.681233    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"]
Nov 03 00:48:25.712147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:25.712098    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768cnvh4"]
Nov 03 00:48:26.283280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.283229    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 03 00:48:26.283553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.283528    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 03 00:48:26.346284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.344741    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"]
Nov 03 00:48:26.346284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.346175    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6ae45849-4c5d-40d0-bb5f-f78a448adbcb" containerName="payload-processing"
Nov 03 00:48:26.346284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.346198    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae45849-4c5d-40d0-bb5f-f78a448adbcb" containerName="payload-processing"
Nov 03 00:48:26.346284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.346230    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8be5dc52-85ed-4c07-8b3b-8549fc818b9e" containerName="payload-processing"
Nov 03 00:48:26.346284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.346237    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8be5dc52-85ed-4c07-8b3b-8549fc818b9e" containerName="payload-processing"
Nov 03 00:48:26.346733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.346681    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae45849-4c5d-40d0-bb5f-f78a448adbcb" containerName="payload-processing"
Nov 03 00:48:26.346733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.346705    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8be5dc52-85ed-4c07-8b3b-8549fc818b9e" containerName="payload-processing"
Nov 03 00:48:26.376980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.375026    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"]
Nov 03 00:48:26.376980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.375159    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.423417 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.423384    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-850433-openshift-default-dockercfg-qt766\""
Nov 03 00:48:26.540312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540215    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-envoy\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540283    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-credential-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540372    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-certs\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540469    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-istio-token\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540506    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540640    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06270d58-ffc7-4d70-afac-7f8f99570583-istio-podinfo\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540679    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-data\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540736    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06270d58-ffc7-4d70-afac-7f8f99570583-istiod-ca-cert\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.540899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.540883    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brctk\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-kube-api-access-brctk\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.574621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.574582    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8be5dc52-85ed-4c07-8b3b-8549fc818b9e" path="/var/lib/kubelet/pods/8be5dc52-85ed-4c07-8b3b-8549fc818b9e/volumes"
Nov 03 00:48:26.642629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.642595    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06270d58-ffc7-4d70-afac-7f8f99570583-istio-podinfo\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.642629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.642632    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-data\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.642657    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06270d58-ffc7-4d70-afac-7f8f99570583-istiod-ca-cert\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.642810    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brctk\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-kube-api-access-brctk\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.642876    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-envoy\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.642907    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-credential-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.642963    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-certs\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.643034    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-istio-token\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.643061    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-data\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.643073    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.643366    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.643480    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06270d58-ffc7-4d70-afac-7f8f99570583-istiod-ca-cert\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.643491    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-certs\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.643818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.643732    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-credential-socket\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.645578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.645559    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06270d58-ffc7-4d70-afac-7f8f99570583-istio-podinfo\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.646003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.645980    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-envoy\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.661488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.661466    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-istio-token\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.661488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.661474    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brctk\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-kube-api-access-brctk\") pod \"e2e-extmod-a-850433-openshift-default-7584745dd9-kld27\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") " pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:26.731192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:26.731162    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:27.070078 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:48:27.070010    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06270d58_ffc7_4d70_afac_7f8f99570583.slice/crio-28aadc11698a58fb333bd4352222db5fabfe24f2a2ffec6500dcfd25e2dd4c71 WatchSource:0}: Error finding container 28aadc11698a58fb333bd4352222db5fabfe24f2a2ffec6500dcfd25e2dd4c71: Status 404 returned error can't find the container with id 28aadc11698a58fb333bd4352222db5fabfe24f2a2ffec6500dcfd25e2dd4c71
Nov 03 00:48:27.071741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.071716    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"]
Nov 03 00:48:27.072926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.072891    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:27.073025 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.072957    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:27.073025 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.072986    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:27.321212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.321114    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:27.321903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.321880    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:27.323070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.323046    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:48:27.649707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.649665    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" event={"ID":"06270d58-ffc7-4d70-afac-7f8f99570583","Type":"ContainerStarted","Data":"30cc1f9cc0cbf0dca64e56b307584c7db06fc0fc5a1e1d38c19029e24d90c91d"}
Nov 03 00:48:27.650283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.649733    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" event={"ID":"06270d58-ffc7-4d70-afac-7f8f99570583","Type":"ContainerStarted","Data":"28aadc11698a58fb333bd4352222db5fabfe24f2a2ffec6500dcfd25e2dd4c71"}
Nov 03 00:48:27.695280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.693295    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podStartSLOduration=1.6932734919999999 podStartE2EDuration="1.693273492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:27.691660281 +0000 UTC m=+2545.247012308" watchObservedRunningTime="2026-11-03 00:48:27.693273492 +0000 UTC m=+2545.248625520"
Nov 03 00:48:27.731874 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.731825    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:27.829045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:27.829008    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:48:28.734135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:28.734099    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:48:28.734632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:28.734170    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 00:48:29.732033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:29.731996    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:48:29.732390 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:29.732358    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 00:48:30.732100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:30.732063    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:48:30.732553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:30.732130    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 00:48:31.731806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:31.731772    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:48:31.732040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:31.731827    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 00:48:32.731826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:32.731788    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused" start-of-body=
Nov 03 00:48:32.732221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:32.731847    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": dial tcp 10.134.0.218:15021: connect: connection refused"
Nov 03 00:48:32.863707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:32.863675    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"]
Nov 03 00:48:32.868719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:32.868691    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:32.883586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:32.883554    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-e62935-openshift-default-dockercfg-bs2xk\""
Nov 03 00:48:32.923072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:32.921309    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"]
Nov 03 00:48:33.023290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023183    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-token\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023226    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-certs\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023344    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-data\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023399    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023525    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-envoy\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023628    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcl5d\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-kube-api-access-kcl5d\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023697    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istiod-ca-cert\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023721    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-podinfo\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.023803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.023787    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-credential-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125315    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcl5d\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-kube-api-access-kcl5d\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125391    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istiod-ca-cert\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125422    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-podinfo\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125450    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-credential-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125510    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-token\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125556    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-certs\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125620    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-data\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125697    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.125941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.125796    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-envoy\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.126083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.126058    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-credential-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.126288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.126074    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-certs\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.126288 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.126139    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-data\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.126478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.126290    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istiod-ca-cert\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.126478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.126356    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-socket\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.129081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.129058    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-podinfo\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.131383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.131360    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-envoy\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.148296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.145193    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-token\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.148296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.146811    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcl5d\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-kube-api-access-kcl5d\") pod \"e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") " pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.185524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.185492    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:33.517555 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:48:33.517516    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1273c82d_11d9_4a6a_aff7_a190d9e419d5.slice/crio-dd763c06aa184e60d5eda68a9fbd332a81f78c58061e3493322781335fa92a82 WatchSource:0}: Error finding container dd763c06aa184e60d5eda68a9fbd332a81f78c58061e3493322781335fa92a82: Status 404 returned error can't find the container with id dd763c06aa184e60d5eda68a9fbd332a81f78c58061e3493322781335fa92a82
Nov 03 00:48:33.519428 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.519324    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"]
Nov 03 00:48:33.521080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.521043    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:33.521161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.521122    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:33.521217 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.521164    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:48:33.697997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.697957    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" event={"ID":"1273c82d-11d9-4a6a-aff7-a190d9e419d5","Type":"ContainerStarted","Data":"a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5"}
Nov 03 00:48:33.698186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.698019    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" event={"ID":"1273c82d-11d9-4a6a-aff7-a190d9e419d5","Type":"ContainerStarted","Data":"dd763c06aa184e60d5eda68a9fbd332a81f78c58061e3493322781335fa92a82"}
Nov 03 00:48:33.737755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.737703    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podStartSLOduration=1.7376837109999999 podStartE2EDuration="1.737683711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:48:33.735827829 +0000 UTC m=+2551.291179854" watchObservedRunningTime="2026-11-03 00:48:33.737683711 +0000 UTC m=+2551.293035737"
Nov 03 00:48:33.738347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.737993    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:33.739358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.739331    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:33.740038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:33.740013    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 00:48:34.186244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:34.186209    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:34.191238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:34.191206    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:48:34.191502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:34.191471    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 00:48:35.185818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:35.185782    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:48:35.186206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:35.185841    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 00:48:36.186029 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:36.185996    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:48:36.186514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:36.186048    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 00:48:37.191977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:37.191936    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:48:37.192645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:37.192612    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 00:48:38.186497 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:38.186457    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:48:38.186811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:38.186781    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 00:48:39.186188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:39.186152    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 03 00:48:39.186737 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:39.186210    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 03 00:48:40.191419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:40.191389    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:40.191916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:40.191844    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:40.192636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:48:40.192615    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 00:48:49.897224 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:48:49.897183    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:49:03.324508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.324467    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"]
Nov 03 00:49:03.330048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.330022    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.347702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.347675    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"]
Nov 03 00:49:03.402200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.402162    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfx75\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-kube-api-access-wfx75\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.402413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.402210    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.402465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.402404    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.402465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.402452    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.503655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.503623    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfx75\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-kube-api-access-wfx75\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.503869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.503666    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.503869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.503750    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.503869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.503788    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.504523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.504493    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.504660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.504534    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.507010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.506983    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.522105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.522074    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfx75\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-kube-api-access-wfx75\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:03.684062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:03.665839    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:04.045728 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:04.045693    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"]
Nov 03 00:49:04.045882 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:49:04.045734    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e5f726_db38_4914_b5fd_ef8e92d13a04.slice/crio-38017ad0000d7a64cb485aec0e846d68f0c2c0b4339affb84de1cfff6098cd4b WatchSource:0}: Error finding container 38017ad0000d7a64cb485aec0e846d68f0c2c0b4339affb84de1cfff6098cd4b: Status 404 returned error can't find the container with id 38017ad0000d7a64cb485aec0e846d68f0c2c0b4339affb84de1cfff6098cd4b
Nov 03 00:49:04.982341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:04.982309    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt" event={"ID":"75e5f726-db38-4914-b5fd-ef8e92d13a04","Type":"ContainerStarted","Data":"cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f"}
Nov 03 00:49:04.982769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:04.982363    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt" event={"ID":"75e5f726-db38-4914-b5fd-ef8e92d13a04","Type":"ContainerStarted","Data":"38017ad0000d7a64cb485aec0e846d68f0c2c0b4339affb84de1cfff6098cd4b"}
Nov 03 00:49:04.982769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:04.982402    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:05.024177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:05.024127    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt" podStartSLOduration=2.024112478 podStartE2EDuration="2.024112478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:05.020979274 +0000 UTC m=+2582.576331292" watchObservedRunningTime="2026-11-03 00:49:05.024112478 +0000 UTC m=+2582.579464502"
Nov 03 00:49:06.452199 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:06.452162    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:49:13.575281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.574322    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"]
Nov 03 00:49:13.582337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.582306    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.603987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.603946    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"]
Nov 03 00:49:13.609363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.609333    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"agc-cred-1510b21983f6\""
Nov 03 00:49:13.660070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.659972    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.660070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.660068    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.660358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.660097    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.660358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.660293    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxd7b\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-kube-api-access-sxd7b\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.782008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.770444    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxd7b\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-kube-api-access-sxd7b\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.797276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.782364    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.820279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.810689    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.820279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.810733    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.820279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.802127    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.820279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.812107    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.834532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.823862    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.856023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.855987    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxd7b\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-kube-api-access-sxd7b\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:13.944530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:13.944495    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:14.267642 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:49:14.267613    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca666b1d_bb8c_403a_ad7d_009f8c066ead.slice/crio-2453ddba709b2a24c469e1dff400d9498debb95e2535d4042e98963372ce2ebd WatchSource:0}: Error finding container 2453ddba709b2a24c469e1dff400d9498debb95e2535d4042e98963372ce2ebd: Status 404 returned error can't find the container with id 2453ddba709b2a24c469e1dff400d9498debb95e2535d4042e98963372ce2ebd
Nov 03 00:49:14.268441 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.268416    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"]
Nov 03 00:49:14.518145 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.518069    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"]
Nov 03 00:49:14.523586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.523560    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.541067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.540504    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"agc-cred-35649651d663\""
Nov 03 00:49:14.541067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.540551    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"agc-cred-43a300ff3f2b\""
Nov 03 00:49:14.541067 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.540976    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"]
Nov 03 00:49:14.622180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.622143    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.622630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.622337    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hbdm\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-kube-api-access-8hbdm\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.622630 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.622376    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.622765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.622656    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.723536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.723505    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.723733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.723715    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.723823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.723807    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.723917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.723901    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hbdm\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-kube-api-access-8hbdm\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.724578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.724554    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.724698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.724557    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.726231 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.726210    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.743723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.743704    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hbdm\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-kube-api-access-8hbdm\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:14.842282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:14.842180    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:15.077290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.077216    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg" event={"ID":"ca666b1d-bb8c-403a-ad7d-009f8c066ead","Type":"ContainerStarted","Data":"a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f"}
Nov 03 00:49:15.077290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.077294    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg" event={"ID":"ca666b1d-bb8c-403a-ad7d-009f8c066ead","Type":"ContainerStarted","Data":"2453ddba709b2a24c469e1dff400d9498debb95e2535d4042e98963372ce2ebd"}
Nov 03 00:49:15.077526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.077323    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:15.117015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.116965    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg" podStartSLOduration=2.116952127 podStartE2EDuration="2.116952127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:15.112947132 +0000 UTC m=+2592.668299158" watchObservedRunningTime="2026-11-03 00:49:15.116952127 +0000 UTC m=+2592.672304177"
Nov 03 00:49:15.150082 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:49:15.150055    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66695227_d8ac_4cae_b4b8_b334414a832a.slice/crio-948d1efa42e084ad6ab54df5981ad33c1d1db21963dcd7b3c4f1c1d4eb482720 WatchSource:0}: Error finding container 948d1efa42e084ad6ab54df5981ad33c1d1db21963dcd7b3c4f1c1d4eb482720: Status 404 returned error can't find the container with id 948d1efa42e084ad6ab54df5981ad33c1d1db21963dcd7b3c4f1c1d4eb482720
Nov 03 00:49:15.152475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.152448    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"]
Nov 03 00:49:15.741692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.741657    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"]
Nov 03 00:49:15.745698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.745673    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"]
Nov 03 00:49:15.749129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:15.749109    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:16.116104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:16.116069    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn" event={"ID":"66695227-d8ac-4cae-b4b8-b334414a832a","Type":"ContainerStarted","Data":"6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86"}
Nov 03 00:49:16.116395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:16.116373    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn" event={"ID":"66695227-d8ac-4cae-b4b8-b334414a832a","Type":"ContainerStarted","Data":"948d1efa42e084ad6ab54df5981ad33c1d1db21963dcd7b3c4f1c1d4eb482720"}
Nov 03 00:49:16.116525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:16.116514    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:16.206228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:16.206129    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn" podStartSLOduration=2.206109164 podStartE2EDuration="2.206109164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:16.163573083 +0000 UTC m=+2593.718925107" watchObservedRunningTime="2026-11-03 00:49:16.206109164 +0000 UTC m=+2593.761461189"
Nov 03 00:49:19.898941 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:19.898910    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:49:20.742415 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:20.742371    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb" podUID="c9998f17-d123-47f3-b44d-4b4bc2a6ae57" containerName="payload-processing" containerID="cri-o://2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a" gracePeriod=25
Nov 03 00:49:20.746879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:20.746835    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt" podUID="75e5f726-db38-4914-b5fd-ef8e92d13a04" containerName="payload-processing" containerID="cri-o://cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f" gracePeriod=25
Nov 03 00:49:21.059303 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.054043    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:49:21.092814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.092791    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:21.127760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.127732    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-provider-credentials\") pod \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") "
Nov 03 00:49:21.127943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.127901    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-provider-credentials\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-provider-credentials\") pod \"75e5f726-db38-4914-b5fd-ef8e92d13a04\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") "
Nov 03 00:49:21.128022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128003    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-routing-overlay-volume\") pod \"75e5f726-db38-4914-b5fd-ef8e92d13a04\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") "
Nov 03 00:49:21.128087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128036    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-routing-overlay-volume\") pod \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") "
Nov 03 00:49:21.128372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128338    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-plugins-config-volume\") pod \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") "
Nov 03 00:49:21.128502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128412    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-plugins-config-volume\") pod \"75e5f726-db38-4914-b5fd-ef8e92d13a04\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") "
Nov 03 00:49:21.128502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128470    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-kube-api-access-5k5gw\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-kube-api-access-5k5gw\") pod \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\" (UID: \"c9998f17-d123-47f3-b44d-4b4bc2a6ae57\") "
Nov 03 00:49:21.128502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128498    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-kube-api-access-wfx75\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-kube-api-access-wfx75\") pod \"75e5f726-db38-4914-b5fd-ef8e92d13a04\" (UID: \"75e5f726-db38-4914-b5fd-ef8e92d13a04\") "
Nov 03 00:49:21.128804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128409    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-routing-overlay-volume" pod "c9998f17-d123-47f3-b44d-4b4bc2a6ae57" (UID: "c9998f17-d123-47f3-b44d-4b4bc2a6ae57"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:21.128804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128416    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-routing-overlay-volume" pod "75e5f726-db38-4914-b5fd-ef8e92d13a04" (UID: "75e5f726-db38-4914-b5fd-ef8e92d13a04"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:21.128952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128795    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-plugins-config-volume" pod "75e5f726-db38-4914-b5fd-ef8e92d13a04" (UID: "75e5f726-db38-4914-b5fd-ef8e92d13a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:21.128952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.128776    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-plugins-config-volume" pod "c9998f17-d123-47f3-b44d-4b4bc2a6ae57" (UID: "c9998f17-d123-47f3-b44d-4b4bc2a6ae57"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:21.129244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.129205    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.129244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.129234    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.129412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.129249    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.129412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.129283    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e5f726-db38-4914-b5fd-ef8e92d13a04-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.131760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.131733    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-provider-credentials" pod "75e5f726-db38-4914-b5fd-ef8e92d13a04" (UID: "75e5f726-db38-4914-b5fd-ef8e92d13a04"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:21.131923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.131896    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-kube-api-access-5k5gw" pod "c9998f17-d123-47f3-b44d-4b4bc2a6ae57" (UID: "c9998f17-d123-47f3-b44d-4b4bc2a6ae57"). InnerVolumeSpecName "kube-api-access-5k5gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:21.132413 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.132387    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-kube-api-access-wfx75" pod "75e5f726-db38-4914-b5fd-ef8e92d13a04" (UID: "75e5f726-db38-4914-b5fd-ef8e92d13a04"). InnerVolumeSpecName "kube-api-access-wfx75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:21.138929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.138902    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-provider-credentials" pod "c9998f17-d123-47f3-b44d-4b4bc2a6ae57" (UID: "c9998f17-d123-47f3-b44d-4b4bc2a6ae57"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:21.164436 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.164399    6000 generic.go:356] "Generic (PLEG): container finished" podID="75e5f726-db38-4914-b5fd-ef8e92d13a04" containerID="cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f" exitCode=0
Nov 03 00:49:21.164577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.164455    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"
Nov 03 00:49:21.164577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.164495    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt" event={"ID":"75e5f726-db38-4914-b5fd-ef8e92d13a04","Type":"ContainerDied","Data":"cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f"}
Nov 03 00:49:21.164577 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.164565    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt" event={"ID":"75e5f726-db38-4914-b5fd-ef8e92d13a04","Type":"ContainerDied","Data":"38017ad0000d7a64cb485aec0e846d68f0c2c0b4339affb84de1cfff6098cd4b"}
Nov 03 00:49:21.164742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.164591    6000 scope.go:122] "RemoveContainer" containerID="cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f"
Nov 03 00:49:21.166382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.166358    6000 generic.go:356] "Generic (PLEG): container finished" podID="c9998f17-d123-47f3-b44d-4b4bc2a6ae57" containerID="2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a" exitCode=0
Nov 03 00:49:21.166523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.166491    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb" event={"ID":"c9998f17-d123-47f3-b44d-4b4bc2a6ae57","Type":"ContainerDied","Data":"2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a"}
Nov 03 00:49:21.166588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.166532    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb" event={"ID":"c9998f17-d123-47f3-b44d-4b4bc2a6ae57","Type":"ContainerDied","Data":"61cbea395a8a687456dc3d4345345ce952b16cbc8ddf13e779b1c61c6e90f6d8"}
Nov 03 00:49:21.166781 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.166767    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"
Nov 03 00:49:21.179200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.179181    6000 scope.go:122] "RemoveContainer" containerID="cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f"
Nov 03 00:49:21.179540 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:21.179510    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f\": container with ID starting with cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f not found: ID does not exist" containerID="cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f"
Nov 03 00:49:21.179613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.179547    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f"} err="failed to get container status \"cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f\": rpc error: code = NotFound desc = could not find container \"cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f\": container with ID starting with cb2fea261904597f551974a071252a4b12b6a19e52d02495ede4bc5e5537f42f not found: ID does not exist"
Nov 03 00:49:21.179613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.179566    6000 scope.go:122] "RemoveContainer" containerID="2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a"
Nov 03 00:49:21.189636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.189617    6000 scope.go:122] "RemoveContainer" containerID="2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a"
Nov 03 00:49:21.189917 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:21.189898    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a\": container with ID starting with 2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a not found: ID does not exist" containerID="2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a"
Nov 03 00:49:21.190022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.189922    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a"} err="failed to get container status \"2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a\": rpc error: code = NotFound desc = could not find container \"2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a\": container with ID starting with 2c6ff4f4e2ca6a38a67e2d713c9bc7141ff77ba436c213cba466cdd6a636a53a not found: ID does not exist"
Nov 03 00:49:21.228720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.228693    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"]
Nov 03 00:49:21.230771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.230740    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k5gw\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-kube-api-access-5k5gw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.230771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.230767    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfx75\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-kube-api-access-wfx75\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.230942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.230777    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c9998f17-d123-47f3-b44d-4b4bc2a6ae57-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.230942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.230788    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/75e5f726-db38-4914-b5fd-ef8e92d13a04-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:21.233244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.233218    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-7768ftbgt"]
Nov 03 00:49:21.264459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.264395    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"]
Nov 03 00:49:21.268081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:21.268059    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-5795th6jb"]
Nov 03 00:49:22.464767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:22.464729    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"]
Nov 03 00:49:22.570571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:22.570544    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e5f726-db38-4914-b5fd-ef8e92d13a04" path="/var/lib/kubelet/pods/75e5f726-db38-4914-b5fd-ef8e92d13a04/volumes"
Nov 03 00:49:22.570973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:22.570959    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9998f17-d123-47f3-b44d-4b4bc2a6ae57" path="/var/lib/kubelet/pods/c9998f17-d123-47f3-b44d-4b4bc2a6ae57/volumes"
Nov 03 00:49:26.119194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:26.119165    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:49:26.191462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:26.191425    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"]
Nov 03 00:49:27.465950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.465913    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c" podUID="bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" containerName="payload-processing" containerID="cri-o://d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e" gracePeriod=25
Nov 03 00:49:27.771971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.771944    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:49:27.876516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.876484    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-kube-api-access-p9z8h\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-kube-api-access-p9z8h\") pod \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") "
Nov 03 00:49:27.876704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.876599    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-provider-credentials\") pod \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") "
Nov 03 00:49:27.876704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.876668    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-plugins-config-volume\") pod \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") "
Nov 03 00:49:27.876803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.876724    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-routing-overlay-volume\") pod \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\" (UID: \"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d\") "
Nov 03 00:49:27.877092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.877062    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-routing-overlay-volume" pod "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" (UID: "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:27.877249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.877091    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-plugins-config-volume" pod "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" (UID: "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:27.877600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.877579    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:27.877640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.877608    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:27.879001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.878975    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-kube-api-access-p9z8h" pod "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" (UID: "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d"). InnerVolumeSpecName "kube-api-access-p9z8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:27.879099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.879017    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-provider-credentials" pod "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" (UID: "bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:27.979602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.979563    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9z8h\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-kube-api-access-p9z8h\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:27.979602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:27.979595    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:28.132571 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.132539    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:49:28.206897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.206853    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"]
Nov 03 00:49:28.246137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.246105    6000 generic.go:356] "Generic (PLEG): container finished" podID="bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" containerID="d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e" exitCode=0
Nov 03 00:49:28.246363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.246172    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"
Nov 03 00:49:28.246363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.246207    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c" event={"ID":"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d","Type":"ContainerDied","Data":"d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e"}
Nov 03 00:49:28.246363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.246294    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c" event={"ID":"bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d","Type":"ContainerDied","Data":"f06d1f25c8c40ae34cda73a9d99d609fc64131447c77651539bf9475285f3cd6"}
Nov 03 00:49:28.246530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.246359    6000 scope.go:122] "RemoveContainer" containerID="d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e"
Nov 03 00:49:28.265181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.264137    6000 scope.go:122] "RemoveContainer" containerID="d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e"
Nov 03 00:49:28.265181 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:28.264616    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e\": container with ID starting with d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e not found: ID does not exist" containerID="d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e"
Nov 03 00:49:28.265181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.264658    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e"} err="failed to get container status \"d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e\": rpc error: code = NotFound desc = could not find container \"d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e\": container with ID starting with d7d16c81c5261b12a1f4b5908705843ad18e94883e15863d1537a85c339e783e not found: ID does not exist"
Nov 03 00:49:28.308327 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.308299    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"]
Nov 03 00:49:28.320920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.320886    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-8cfckvz9c"]
Nov 03 00:49:28.569351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:28.569248    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" path="/var/lib/kubelet/pods/bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d/volumes"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.934426    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"]
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935305    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935320    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935352    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c9998f17-d123-47f3-b44d-4b4bc2a6ae57" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935357    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9998f17-d123-47f3-b44d-4b4bc2a6ae57" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935366    6000 cpu_manager.go:414] "containerMap: removing container" podUID="75e5f726-db38-4914-b5fd-ef8e92d13a04" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935371    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e5f726-db38-4914-b5fd-ef8e92d13a04" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935484    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc46b83a-34ec-4f92-a8fb-0ec63ec09a0d" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935498    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9998f17-d123-47f3-b44d-4b4bc2a6ae57" containerName="payload-processing"
Nov 03 00:49:30.936289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.935506    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e5f726-db38-4914-b5fd-ef8e92d13a04" containerName="payload-processing"
Nov 03 00:49:30.940972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.940940    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:30.958307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.958247    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"]
Nov 03 00:49:30.965788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.965763    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 00:49:30.965788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.965767    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 00:49:30.965982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.965798    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 00:49:30.965982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.965798    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 00:49:30.966291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.966247    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-61d262258feb\""
Nov 03 00:49:30.966291 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.966250    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-99029648dbea\""
Nov 03 00:49:30.966475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:30.966326    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-psmj8\""
Nov 03 00:49:31.040488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.040448    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-plugins-config-volume\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.040700 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.040576    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-kube-api-access-7mbt8\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.040700 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.040674    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-routing-overlay-volume\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.040829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.040793    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-provider-credentials\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.141977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.141941    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-provider-credentials\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.142155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.142061    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-plugins-config-volume\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.142155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.142114    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-kube-api-access-7mbt8\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.142233 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.142176    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-routing-overlay-volume\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.142924 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.142896    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-routing-overlay-volume\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.143073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.142958    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-plugins-config-volume\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.145082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.145037    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-provider-credentials\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.161777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.161748    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-kube-api-access-7mbt8\") pod \"payload-processing-external-model-76d5d774bc-r25fn\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") " pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.203942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.194851    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg" podUID="abd8bca4-d096-4296-971e-015d9e4b8017" containerName="payload-processing" containerID="cri-o://ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69" gracePeriod=25
Nov 03 00:49:31.257028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.256997    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:31.515427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.515402    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:49:31.566246 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:49:31.566214    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804e2180_7acf_4231_af26_c3e6ff11ff76.slice/crio-e4d6e1e889b20c694c240cdc1f1c1ca1990735a4c59c466520b6829d83a78c93 WatchSource:0}: Error finding container e4d6e1e889b20c694c240cdc1f1c1ca1990735a4c59c466520b6829d83a78c93: Status 404 returned error can't find the container with id e4d6e1e889b20c694c240cdc1f1c1ca1990735a4c59c466520b6829d83a78c93
Nov 03 00:49:31.566615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.566591    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"]
Nov 03 00:49:31.652890 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.652859    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-plugins-config-volume\") pod \"abd8bca4-d096-4296-971e-015d9e4b8017\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") "
Nov 03 00:49:31.652990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.652925    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-kube-api-access-8247p\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-kube-api-access-8247p\") pod \"abd8bca4-d096-4296-971e-015d9e4b8017\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") "
Nov 03 00:49:31.652990 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.652949    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-routing-overlay-volume\") pod \"abd8bca4-d096-4296-971e-015d9e4b8017\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") "
Nov 03 00:49:31.653086 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.653051    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-provider-credentials\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-provider-credentials\") pod \"abd8bca4-d096-4296-971e-015d9e4b8017\" (UID: \"abd8bca4-d096-4296-971e-015d9e4b8017\") "
Nov 03 00:49:31.653658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.653633    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-routing-overlay-volume" pod "abd8bca4-d096-4296-971e-015d9e4b8017" (UID: "abd8bca4-d096-4296-971e-015d9e4b8017"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:31.653774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.653728    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-plugins-config-volume" pod "abd8bca4-d096-4296-971e-015d9e4b8017" (UID: "abd8bca4-d096-4296-971e-015d9e4b8017"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:31.654328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.654303    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:31.654426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.654332    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/abd8bca4-d096-4296-971e-015d9e4b8017-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:31.655648 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.655624    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-provider-credentials" pod "abd8bca4-d096-4296-971e-015d9e4b8017" (UID: "abd8bca4-d096-4296-971e-015d9e4b8017"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:31.656568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.656544    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-kube-api-access-8247p" pod "abd8bca4-d096-4296-971e-015d9e4b8017" (UID: "abd8bca4-d096-4296-971e-015d9e4b8017"). InnerVolumeSpecName "kube-api-access-8247p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:31.755662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.755587    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8247p\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-kube-api-access-8247p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:31.755662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:31.755616    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/abd8bca4-d096-4296-971e-015d9e4b8017-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:32.280681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.280644    6000 generic.go:356] "Generic (PLEG): container finished" podID="abd8bca4-d096-4296-971e-015d9e4b8017" containerID="ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69" exitCode=0
Nov 03 00:49:32.281208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.280766    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg" event={"ID":"abd8bca4-d096-4296-971e-015d9e4b8017","Type":"ContainerDied","Data":"ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69"}
Nov 03 00:49:32.281208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.280784    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"
Nov 03 00:49:32.281208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.280833    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg" event={"ID":"abd8bca4-d096-4296-971e-015d9e4b8017","Type":"ContainerDied","Data":"6e1dc21f0729a72b567da1fe3bb1367c9278edf409f663bbb94ea310cb7e66e8"}
Nov 03 00:49:32.281208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.280885    6000 scope.go:122] "RemoveContainer" containerID="ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69"
Nov 03 00:49:32.283180 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.283153    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn" event={"ID":"804e2180-7acf-4231-af26-c3e6ff11ff76","Type":"ContainerStarted","Data":"50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe"}
Nov 03 00:49:32.283328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.283206    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn" event={"ID":"804e2180-7acf-4231-af26-c3e6ff11ff76","Type":"ContainerStarted","Data":"e4d6e1e889b20c694c240cdc1f1c1ca1990735a4c59c466520b6829d83a78c93"}
Nov 03 00:49:32.283328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.283242    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:32.292081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.292067    6000 scope.go:122] "RemoveContainer" containerID="ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69"
Nov 03 00:49:32.292356 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:32.292336    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69\": container with ID starting with ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69 not found: ID does not exist" containerID="ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69"
Nov 03 00:49:32.292416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.292364    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69"} err="failed to get container status \"ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69\": rpc error: code = NotFound desc = could not find container \"ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69\": container with ID starting with ee044f19fcfae533e660ffa560467984711ff5feee2de18eb75390d400b86a69 not found: ID does not exist"
Nov 03 00:49:32.331296 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.331268    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"]
Nov 03 00:49:32.343332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.343307    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5f57g9lwg"]
Nov 03 00:49:32.372844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.372804    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn" podStartSLOduration=2.37279208 podStartE2EDuration="2.37279208s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:32.370318968 +0000 UTC m=+2609.925670993" watchObservedRunningTime="2026-11-03 00:49:32.37279208 +0000 UTC m=+2609.928144120"
Nov 03 00:49:32.583540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:32.583435    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd8bca4-d096-4296-971e-015d9e4b8017" path="/var/lib/kubelet/pods/abd8bca4-d096-4296-971e-015d9e4b8017/volumes"
Nov 03 00:49:33.207202 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.207148    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp" podUID="40e5f52c-9d0b-44dd-a4e5-cffb398588f2" containerName="payload-processing" containerID="cri-o://834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6" gracePeriod=25
Nov 03 00:49:33.284089 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:33.284053    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e5f52c_9d0b_44dd_a4e5_cffb398588f2.slice/crio-834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40e5f52c_9d0b_44dd_a4e5_cffb398588f2.slice/crio-conmon-834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 00:49:33.489013 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.488986    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:49:33.582790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.582759    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-plugins-config-volume\") pod \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") "
Nov 03 00:49:33.582790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.582795    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-kube-api-access-m5jtk\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-kube-api-access-m5jtk\") pod \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") "
Nov 03 00:49:33.583039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.582815    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-routing-overlay-volume\") pod \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") "
Nov 03 00:49:33.583039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.582885    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-provider-credentials\") pod \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\" (UID: \"40e5f52c-9d0b-44dd-a4e5-cffb398588f2\") "
Nov 03 00:49:33.583289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.583228    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-plugins-config-volume" pod "40e5f52c-9d0b-44dd-a4e5-cffb398588f2" (UID: "40e5f52c-9d0b-44dd-a4e5-cffb398588f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:33.583397 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.583298    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-routing-overlay-volume" pod "40e5f52c-9d0b-44dd-a4e5-cffb398588f2" (UID: "40e5f52c-9d0b-44dd-a4e5-cffb398588f2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:49:33.583599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.583577    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:33.583701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.583605    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:33.585118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.585096    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-kube-api-access-m5jtk" pod "40e5f52c-9d0b-44dd-a4e5-cffb398588f2" (UID: "40e5f52c-9d0b-44dd-a4e5-cffb398588f2"). InnerVolumeSpecName "kube-api-access-m5jtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:33.585313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.585292    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-provider-credentials" pod "40e5f52c-9d0b-44dd-a4e5-cffb398588f2" (UID: "40e5f52c-9d0b-44dd-a4e5-cffb398588f2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:49:33.685316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.685286    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:33.685316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:33.685312    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5jtk\" (UniqueName: \"kubernetes.io/projected/40e5f52c-9d0b-44dd-a4e5-cffb398588f2-kube-api-access-m5jtk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:49:34.303440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.303407    6000 generic.go:356] "Generic (PLEG): container finished" podID="40e5f52c-9d0b-44dd-a4e5-cffb398588f2" containerID="834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6" exitCode=0
Nov 03 00:49:34.303929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.303463    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"
Nov 03 00:49:34.303929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.303486    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp" event={"ID":"40e5f52c-9d0b-44dd-a4e5-cffb398588f2","Type":"ContainerDied","Data":"834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6"}
Nov 03 00:49:34.303929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.303537    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp" event={"ID":"40e5f52c-9d0b-44dd-a4e5-cffb398588f2","Type":"ContainerDied","Data":"64179c2f575e62e591dd64c1f89a531ba21f7a2d89c2814193cce8cbffd11413"}
Nov 03 00:49:34.303929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.303560    6000 scope.go:122] "RemoveContainer" containerID="834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6"
Nov 03 00:49:34.317733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.317699    6000 scope.go:122] "RemoveContainer" containerID="834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6"
Nov 03 00:49:34.318035 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:34.318015    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6\": container with ID starting with 834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6 not found: ID does not exist" containerID="834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6"
Nov 03 00:49:34.318118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.318044    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6"} err="failed to get container status \"834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6\": rpc error: code = NotFound desc = could not find container \"834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6\": container with ID starting with 834719be97671ce06a11164ccde3b75211e0dec8bd38bf044e396401da543ae6 not found: ID does not exist"
Nov 03 00:49:34.353858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.353827    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"]
Nov 03 00:49:34.365026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.365000    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784rlltp"]
Nov 03 00:49:34.569326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:34.569218    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40e5f52c-9d0b-44dd-a4e5-cffb398588f2" path="/var/lib/kubelet/pods/40e5f52c-9d0b-44dd-a4e5-cffb398588f2/volumes"
Nov 03 00:49:38.465044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.465009    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"]
Nov 03 00:49:38.466388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.466330    6000 cpu_manager.go:414] "containerMap: removing container" podUID="40e5f52c-9d0b-44dd-a4e5-cffb398588f2" containerName="payload-processing"
Nov 03 00:49:38.466388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.466355    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40e5f52c-9d0b-44dd-a4e5-cffb398588f2" containerName="payload-processing"
Nov 03 00:49:38.466388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.466375    6000 cpu_manager.go:414] "containerMap: removing container" podUID="abd8bca4-d096-4296-971e-015d9e4b8017" containerName="payload-processing"
Nov 03 00:49:38.466388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.466382    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd8bca4-d096-4296-971e-015d9e4b8017" containerName="payload-processing"
Nov 03 00:49:38.466829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.466585    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="40e5f52c-9d0b-44dd-a4e5-cffb398588f2" containerName="payload-processing"
Nov 03 00:49:38.466829 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.466601    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd8bca4-d096-4296-971e-015d9e4b8017" containerName="payload-processing"
Nov 03 00:49:38.472161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.472130    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.492897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.492873    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"]
Nov 03 00:49:38.565320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565246    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565410    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565524 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565463    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h878x\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-kube-api-access-h878x\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565535    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565603    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565685    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565716    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565756    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.565812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.565798    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667136 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667104    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667183    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h878x\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-kube-api-access-h878x\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667213    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667246    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667437    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667509    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667583    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667613    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.667788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667727    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.668431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667789    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.668431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.667996    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.668431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.668122    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.668652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.668599    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.668870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.668847    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.670774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.670752    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.670993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.670974    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.688032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.688003    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.688190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.688132    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h878x\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-kube-api-access-h878x\") pod \"maas-default-gateway-openshift-default-6cc458d887-w5m48\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:38.788079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:38.787986    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:39.104885 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:49:39.104855    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a1e7b0f_2b36_4171_8233_395051e2fb8d.slice/crio-77820c775d0023d6746aaedcbb11d71162afb5f28515918566cd66ccc6591366 WatchSource:0}: Error finding container 77820c775d0023d6746aaedcbb11d71162afb5f28515918566cd66ccc6591366: Status 404 returned error can't find the container with id 77820c775d0023d6746aaedcbb11d71162afb5f28515918566cd66ccc6591366
Nov 03 00:49:39.105596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.105570    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"]
Nov 03 00:49:39.107702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.107671    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:49:39.107771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.107735    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:49:39.107771 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.107765    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:49:39.379196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.379092    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" event={"ID":"0a1e7b0f-2b36-4171-8233-395051e2fb8d","Type":"ContainerStarted","Data":"afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a"}
Nov 03 00:49:39.379196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.379158    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" event={"ID":"0a1e7b0f-2b36-4171-8233-395051e2fb8d","Type":"ContainerStarted","Data":"77820c775d0023d6746aaedcbb11d71162afb5f28515918566cd66ccc6591366"}
Nov 03 00:49:39.419733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.419683    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podStartSLOduration=1.419667304 podStartE2EDuration="1.419667304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:39.418121844 +0000 UTC m=+2616.973473868" watchObservedRunningTime="2026-11-03 00:49:39.419667304 +0000 UTC m=+2616.975019326"
Nov 03 00:49:39.788515 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:39.788436    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:40.800282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:40.799133    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:49:40.800282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:40.799199    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 00:49:41.795693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:41.795654    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:49:41.795996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:41.795964    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 00:49:42.805281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:42.797656    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:49:42.805281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:42.797737    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 00:49:43.326280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:43.323493    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:49:43.789282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:43.788696    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:49:43.789282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:43.788747    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 00:49:44.794281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:44.793652    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:49:44.794281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:44.793715    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 00:49:45.788677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:45.788633    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:49:45.788886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:45.788712    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 00:49:46.788588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:46.788547    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 03 00:49:46.789003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:46.788627    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 03 00:49:48.270989 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:48.270958    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:48.271528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:48.271506    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:48.272420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:48.272392    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 00:49:49.902526 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:49:49.902481    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:49:52.356389 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.356357    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"]
Nov 03 00:49:52.362639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.362616    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.378285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.378225    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"agc-cred-f5e831b1aab5\""
Nov 03 00:49:52.378595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.378574    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"]
Nov 03 00:49:52.477697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.477664    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.477875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.477711    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6hd2\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-kube-api-access-d6hd2\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.477875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.477830    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.477950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.477877    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.579503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.579473    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.579698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.579516    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6hd2\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-kube-api-access-d6hd2\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.579698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.579563    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.579698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.579594    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.580153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.580120    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.580392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.580369    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.582304 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.582282    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.598419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.598383    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6hd2\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-kube-api-access-d6hd2\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.678339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.678243    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:52.980408 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:49:52.980375    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b88892_130c_47a9_9ac8_cf656e6095dc.slice/crio-1fefe66290f65fc0352d95508b5359970b87e90b43eddd46c4fb294b9c7cb792 WatchSource:0}: Error finding container 1fefe66290f65fc0352d95508b5359970b87e90b43eddd46c4fb294b9c7cb792: Status 404 returned error can't find the container with id 1fefe66290f65fc0352d95508b5359970b87e90b43eddd46c4fb294b9c7cb792
Nov 03 00:49:52.981560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:52.981530    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"]
Nov 03 00:49:53.554795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:53.554757    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8" event={"ID":"f4b88892-130c-47a9-9ac8-cf656e6095dc","Type":"ContainerStarted","Data":"5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277"}
Nov 03 00:49:53.555339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:53.554811    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8" event={"ID":"f4b88892-130c-47a9-9ac8-cf656e6095dc","Type":"ContainerStarted","Data":"1fefe66290f65fc0352d95508b5359970b87e90b43eddd46c4fb294b9c7cb792"}
Nov 03 00:49:53.555339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:53.554846    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:49:53.592746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:49:53.592694    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8" podStartSLOduration=1.5926800239999999 podStartE2EDuration="1.592680024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:49:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:49:53.590093444 +0000 UTC m=+2631.145445470" watchObservedRunningTime="2026-11-03 00:49:53.592680024 +0000 UTC m=+2631.148032060"
Nov 03 00:50:00.231607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.231566    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"]
Nov 03 00:50:00.236463 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.236438    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.254004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.253972    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"]
Nov 03 00:50:00.366198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.366146    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.366465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.366311    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn4h2\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-kube-api-access-kn4h2\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.366465 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.366383    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.366625 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.366601    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.479281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.477717    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn4h2\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-kube-api-access-kn4h2\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.479281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.477817    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.479281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.477982    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.479281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.478084    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.480699 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.480667    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.481930 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.481793    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.516765 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.516724    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.555913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.545382    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn4h2\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-kube-api-access-kn4h2\") pod \"payload-processing-external-model-5d5f4bbcf7-mpcdh\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.555913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.554753    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:00.984464 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:50:00.984432    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475cfeb5_e533_469e_b952_0b8cd36f0f82.slice/crio-052deee154082a4c4e34027565899419c8788cd30646ddc54d954968f14686e7 WatchSource:0}: Error finding container 052deee154082a4c4e34027565899419c8788cd30646ddc54d954968f14686e7: Status 404 returned error can't find the container with id 052deee154082a4c4e34027565899419c8788cd30646ddc54d954968f14686e7
Nov 03 00:50:00.984736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:00.984711    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"]
Nov 03 00:50:01.722281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:01.721773    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh" event={"ID":"475cfeb5-e533-469e-b952-0b8cd36f0f82","Type":"ContainerStarted","Data":"3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f"}
Nov 03 00:50:01.722281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:01.721830    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh" event={"ID":"475cfeb5-e533-469e-b952-0b8cd36f0f82","Type":"ContainerStarted","Data":"052deee154082a4c4e34027565899419c8788cd30646ddc54d954968f14686e7"}
Nov 03 00:50:01.722893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:01.722610    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:01.775222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:01.775167    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh" podStartSLOduration=1.775153656 podStartE2EDuration="1.775153656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:01.772909714 +0000 UTC m=+2639.328261825" watchObservedRunningTime="2026-11-03 00:50:01.775153656 +0000 UTC m=+2639.330505679"
Nov 03 00:50:03.524799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:03.524761    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"]
Nov 03 00:50:04.571107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:04.571079    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:50:04.643988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:04.643466    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"]
Nov 03 00:50:08.694189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.694147    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"]
Nov 03 00:50:08.704752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.704714    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"]
Nov 03 00:50:08.704910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.704814    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.738178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.738139    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh" podUID="475cfeb5-e533-469e-b952-0b8cd36f0f82" containerName="payload-processing" containerID="cri-o://3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f" gracePeriod=25
Nov 03 00:50:08.785557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.785526    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"]
Nov 03 00:50:08.791007 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.790986    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:08.807094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.807064    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"]
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.921774    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.921831    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.921884    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2kg6\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-kube-api-access-d2kg6\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.921939    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922001    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922060    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922115    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922186    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922223    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922348    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922391    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlc2r\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-kube-api-access-jlc2r\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922550    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:08.945333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:08.922591    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.023943    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.024006    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.024107    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.024148    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlc2r\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-kube-api-access-jlc2r\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025462    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025584    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025628    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025659    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025684    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025737    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2kg6\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-kube-api-access-d2kg6\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025807    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025870    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025927    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.026574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.025987    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.027746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.027040    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.027746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.027198    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.028795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.028600    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.028928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.028832    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.031131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.029047    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.031131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.029294    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.044318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.036763    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.044318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.038579    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.044318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.041665    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.045060 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.045015    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlc2r\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-kube-api-access-jlc2r\") pod \"payload-processing-external-model-5d5f4bbcf7-2ngxx\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.048409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.048379    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.048747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.048700    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2kg6\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-kube-api-access-d2kg6\") pod \"maas-default-gateway-openshift-default-6cc458d887-zqd94\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.134583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.134539    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:09.315807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.315244    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:09.326696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.326669    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:09.434063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.434028    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-kube-api-access-kn4h2\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-kube-api-access-kn4h2\") pod \"475cfeb5-e533-469e-b952-0b8cd36f0f82\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") "
Nov 03 00:50:09.434272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.434154    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-plugins-config-volume\") pod \"475cfeb5-e533-469e-b952-0b8cd36f0f82\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") "
Nov 03 00:50:09.434364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.434273    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-provider-credentials\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-provider-credentials\") pod \"475cfeb5-e533-469e-b952-0b8cd36f0f82\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") "
Nov 03 00:50:09.434364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.434311    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-routing-overlay-volume\") pod \"475cfeb5-e533-469e-b952-0b8cd36f0f82\" (UID: \"475cfeb5-e533-469e-b952-0b8cd36f0f82\") "
Nov 03 00:50:09.435010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.434961    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-plugins-config-volume" pod "475cfeb5-e533-469e-b952-0b8cd36f0f82" (UID: "475cfeb5-e533-469e-b952-0b8cd36f0f82"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:09.435426 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.435395    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-routing-overlay-volume" pod "475cfeb5-e533-469e-b952-0b8cd36f0f82" (UID: "475cfeb5-e533-469e-b952-0b8cd36f0f82"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:09.437870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.437815    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:09.437870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.437848    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/475cfeb5-e533-469e-b952-0b8cd36f0f82-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:09.438177 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.438149    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-kube-api-access-kn4h2" pod "475cfeb5-e533-469e-b952-0b8cd36f0f82" (UID: "475cfeb5-e533-469e-b952-0b8cd36f0f82"). InnerVolumeSpecName "kube-api-access-kn4h2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:09.440943 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.440911    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-provider-credentials" pod "475cfeb5-e533-469e-b952-0b8cd36f0f82" (UID: "475cfeb5-e533-469e-b952-0b8cd36f0f82"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:09.539031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.538948    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kn4h2\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-kube-api-access-kn4h2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:09.539031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.538979    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/475cfeb5-e533-469e-b952-0b8cd36f0f82-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:09.544839 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:50:09.544809    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4f6cc13_e393_4464_8016_e633c728a5b5.slice/crio-350d34370b3c58b8fb74ebe0f99f35e37adf32465c68169b4a10f361e15b6a77 WatchSource:0}: Error finding container 350d34370b3c58b8fb74ebe0f99f35e37adf32465c68169b4a10f361e15b6a77: Status 404 returned error can't find the container with id 350d34370b3c58b8fb74ebe0f99f35e37adf32465c68169b4a10f361e15b6a77
Nov 03 00:50:09.545480 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.545416    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"]
Nov 03 00:50:09.644187 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.644124    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml" podUID="fbad7667-36ff-4064-9293-6d5499f28812" containerName="payload-processing" containerID="cri-o://8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e" gracePeriod=25
Nov 03 00:50:09.659583 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.659552    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"]
Nov 03 00:50:09.673153 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:50:09.673122    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f4fb1a_2c68_49ce_9cc6_8c18e4949548.slice/crio-7030151cf4ba1ed22330136dc8a889f559095ca7fcb20652e53059cc5de12b05 WatchSource:0}: Error finding container 7030151cf4ba1ed22330136dc8a889f559095ca7fcb20652e53059cc5de12b05: Status 404 returned error can't find the container with id 7030151cf4ba1ed22330136dc8a889f559095ca7fcb20652e53059cc5de12b05
Nov 03 00:50:09.765536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.765474    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:50:09.766118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.766088    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:50:09.766304 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.766283    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:50:09.917101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.917065    6000 generic.go:356] "Generic (PLEG): container finished" podID="475cfeb5-e533-469e-b952-0b8cd36f0f82" containerID="3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f" exitCode=0
Nov 03 00:50:09.917510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.917489    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh" event={"ID":"475cfeb5-e533-469e-b952-0b8cd36f0f82","Type":"ContainerDied","Data":"3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f"}
Nov 03 00:50:09.917668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.917655    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh" event={"ID":"475cfeb5-e533-469e-b952-0b8cd36f0f82","Type":"ContainerDied","Data":"052deee154082a4c4e34027565899419c8788cd30646ddc54d954968f14686e7"}
Nov 03 00:50:09.917768 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.917758    6000 scope.go:122] "RemoveContainer" containerID="3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f"
Nov 03 00:50:09.918121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:09.918100    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"
Nov 03 00:50:10.015276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.005655    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" event={"ID":"82f4fb1a-2c68-49ce-9cc6-8c18e4949548","Type":"ContainerStarted","Data":"7030151cf4ba1ed22330136dc8a889f559095ca7fcb20652e53059cc5de12b05"}
Nov 03 00:50:10.040409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.040363    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx" event={"ID":"c4f6cc13-e393-4464-8016-e633c728a5b5","Type":"ContainerStarted","Data":"322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc"}
Nov 03 00:50:10.040778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.040752    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx" event={"ID":"c4f6cc13-e393-4464-8016-e633c728a5b5","Type":"ContainerStarted","Data":"350d34370b3c58b8fb74ebe0f99f35e37adf32465c68169b4a10f361e15b6a77"}
Nov 03 00:50:10.041280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.041205    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:10.054051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.053764    6000 scope.go:122] "RemoveContainer" containerID="3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f"
Nov 03 00:50:10.054466 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:50:10.054195    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f\": container with ID starting with 3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f not found: ID does not exist" containerID="3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f"
Nov 03 00:50:10.054466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.054265    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f"} err="failed to get container status \"3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f\": rpc error: code = NotFound desc = could not find container \"3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f\": container with ID starting with 3bf9014a93882446cf84db9cba125374e1c158f2c768519d06a9acb051f00b7f not found: ID does not exist"
Nov 03 00:50:10.078383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.078296    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"]
Nov 03 00:50:10.089638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.089607    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-mpcdh"]
Nov 03 00:50:10.115386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.115327    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx" podStartSLOduration=2.115307896 podStartE2EDuration="2.115307896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:10.113146158 +0000 UTC m=+2647.668498192" watchObservedRunningTime="2026-11-03 00:50:10.115307896 +0000 UTC m=+2647.670659921"
Nov 03 00:50:10.197112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.197084    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:50:10.291951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.291914    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-provider-credentials\") pod \"fbad7667-36ff-4064-9293-6d5499f28812\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") "
Nov 03 00:50:10.292137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.292003    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-plugins-config-volume\") pod \"fbad7667-36ff-4064-9293-6d5499f28812\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") "
Nov 03 00:50:10.292137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.292093    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-routing-overlay-volume\") pod \"fbad7667-36ff-4064-9293-6d5499f28812\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") "
Nov 03 00:50:10.292321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.292198    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-kube-api-access-t445r\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-kube-api-access-t445r\") pod \"fbad7667-36ff-4064-9293-6d5499f28812\" (UID: \"fbad7667-36ff-4064-9293-6d5499f28812\") "
Nov 03 00:50:10.293714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.293687    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-plugins-config-volume" pod "fbad7667-36ff-4064-9293-6d5499f28812" (UID: "fbad7667-36ff-4064-9293-6d5499f28812"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:10.294297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.294272    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-routing-overlay-volume" pod "fbad7667-36ff-4064-9293-6d5499f28812" (UID: "fbad7667-36ff-4064-9293-6d5499f28812"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:10.294986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.294962    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-provider-credentials" pod "fbad7667-36ff-4064-9293-6d5499f28812" (UID: "fbad7667-36ff-4064-9293-6d5499f28812"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:10.295916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.295896    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-kube-api-access-t445r" pod "fbad7667-36ff-4064-9293-6d5499f28812" (UID: "fbad7667-36ff-4064-9293-6d5499f28812"). InnerVolumeSpecName "kube-api-access-t445r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:10.394332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.394239    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t445r\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-kube-api-access-t445r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:10.394332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.394333    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbad7667-36ff-4064-9293-6d5499f28812-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:10.394553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.394344    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:10.394553 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.394353    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbad7667-36ff-4064-9293-6d5499f28812-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:10.547955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.547918    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"]
Nov 03 00:50:10.571531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.571503    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="475cfeb5-e533-469e-b952-0b8cd36f0f82" path="/var/lib/kubelet/pods/475cfeb5-e533-469e-b952-0b8cd36f0f82/volumes"
Nov 03 00:50:10.579219 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.579190    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"]
Nov 03 00:50:10.580162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.580140    6000 cpu_manager.go:414] "containerMap: removing container" podUID="475cfeb5-e533-469e-b952-0b8cd36f0f82" containerName="payload-processing"
Nov 03 00:50:10.580162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.580160    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="475cfeb5-e533-469e-b952-0b8cd36f0f82" containerName="payload-processing"
Nov 03 00:50:10.580427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.580203    6000 cpu_manager.go:414] "containerMap: removing container" podUID="fbad7667-36ff-4064-9293-6d5499f28812" containerName="payload-processing"
Nov 03 00:50:10.580427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.580208    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbad7667-36ff-4064-9293-6d5499f28812" containerName="payload-processing"
Nov 03 00:50:10.580427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.580390    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbad7667-36ff-4064-9293-6d5499f28812" containerName="payload-processing"
Nov 03 00:50:10.580427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.580406    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="475cfeb5-e533-469e-b952-0b8cd36f0f82" containerName="payload-processing"
Nov 03 00:50:10.584581 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.584563    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.598678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.598653    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-plugins-config-volume\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.598840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.598813    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4f5l\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-kube-api-access-t4f5l\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.598914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.598895    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-provider-credentials\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.598972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.598958    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-routing-overlay-volume\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.599073 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.599052    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-b86acbfb246d\""
Nov 03 00:50:10.602693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.602673    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"]
Nov 03 00:50:10.710798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.710700    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-routing-overlay-volume\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.712317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.712286    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-plugins-config-volume\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.713587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.713560    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4f5l\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-kube-api-access-t4f5l\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.714218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.714184    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-provider-credentials\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.714978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.711862    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-routing-overlay-volume\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.715143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.713197    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-plugins-config-volume\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.722385 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.722361    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-provider-credentials\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.737115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.737076    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4f5l\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-kube-api-access-t4f5l\") pod \"payload-processing-external-model-5db79bd7f6-9b7f6\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") " pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:10.911873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:10.908987    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:11.053483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.053443    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" event={"ID":"82f4fb1a-2c68-49ce-9cc6-8c18e4949548","Type":"ContainerStarted","Data":"60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4"}
Nov 03 00:50:11.055069 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.055043    6000 generic.go:356] "Generic (PLEG): container finished" podID="fbad7667-36ff-4064-9293-6d5499f28812" containerID="8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e" exitCode=0
Nov 03 00:50:11.055222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.055082    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"
Nov 03 00:50:11.055222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.055126    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml" event={"ID":"fbad7667-36ff-4064-9293-6d5499f28812","Type":"ContainerDied","Data":"8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e"}
Nov 03 00:50:11.055222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.055170    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml" event={"ID":"fbad7667-36ff-4064-9293-6d5499f28812","Type":"ContainerDied","Data":"62604d377a35fa0f66ac300899dac8110e1f676c812a4fe685c80c13b0206cce"}
Nov 03 00:50:11.055222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.055193    6000 scope.go:122] "RemoveContainer" containerID="8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e"
Nov 03 00:50:11.068601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.068558    6000 scope.go:122] "RemoveContainer" containerID="8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e"
Nov 03 00:50:11.068934 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:50:11.068908    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e\": container with ID starting with 8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e not found: ID does not exist" containerID="8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e"
Nov 03 00:50:11.069022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.068947    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e"} err="failed to get container status \"8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e\": rpc error: code = NotFound desc = could not find container \"8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e\": container with ID starting with 8e2eb3558866f3aaf0949b63bc62f3b38ac537fa44ba59da071c01967412867e not found: ID does not exist"
Nov 03 00:50:11.127002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.126936    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podStartSLOduration=3.126917783 podStartE2EDuration="3.126917783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:11.122724601 +0000 UTC m=+2648.678076626" watchObservedRunningTime="2026-11-03 00:50:11.126917783 +0000 UTC m=+2648.682269808"
Nov 03 00:50:11.158938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.158894    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"]
Nov 03 00:50:11.164236 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.164203    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d56pfml"]
Nov 03 00:50:11.218482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.218456    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"]
Nov 03 00:50:11.219412 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:50:11.219386    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod717bd066_0fd1_442a_b476_055cb39bdf30.slice/crio-6b856df9a4c9d6c52266a6f063628d3e4ff32579e86498b3d357a2cd5f2b6794 WatchSource:0}: Error finding container 6b856df9a4c9d6c52266a6f063628d3e4ff32579e86498b3d357a2cd5f2b6794: Status 404 returned error can't find the container with id 6b856df9a4c9d6c52266a6f063628d3e4ff32579e86498b3d357a2cd5f2b6794
Nov 03 00:50:11.327206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.327175    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:11.328740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.328715    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:50:11.328852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:11.328761    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 00:50:12.067070 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:12.067032    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6" event={"ID":"717bd066-0fd1-442a-b476-055cb39bdf30","Type":"ContainerStarted","Data":"1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8"}
Nov 03 00:50:12.067610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:12.067092    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6" event={"ID":"717bd066-0fd1-442a-b476-055cb39bdf30","Type":"ContainerStarted","Data":"6b856df9a4c9d6c52266a6f063628d3e4ff32579e86498b3d357a2cd5f2b6794"}
Nov 03 00:50:12.067610 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:12.067572    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:12.114916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:12.114838    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6" podStartSLOduration=2.114818291 podStartE2EDuration="2.114818291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:12.111673084 +0000 UTC m=+2649.667025107" watchObservedRunningTime="2026-11-03 00:50:12.114818291 +0000 UTC m=+2649.670170317"
Nov 03 00:50:12.327329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:12.327219    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:50:12.327329 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:12.327312    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 00:50:12.570205 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:12.570170    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbad7667-36ff-4064-9293-6d5499f28812" path="/var/lib/kubelet/pods/fbad7667-36ff-4064-9293-6d5499f28812/volumes"
Nov 03 00:50:13.328134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:13.328094    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:50:13.328596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:13.328157    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 00:50:14.328333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:14.328282    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:50:14.328791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:14.328345    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 00:50:15.327816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:15.327781    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:50:15.327986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:15.327839    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 00:50:16.327240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.327193    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:50:16.327777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.327275    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 00:50:16.412653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.412614    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"]
Nov 03 00:50:16.415921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.415894    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:16.447181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.447148    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"]
Nov 03 00:50:16.460486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.459866    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.472770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.472597    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"]
Nov 03 00:50:16.565735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.565703    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb69q\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-kube-api-access-zb69q\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.565914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.565838    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.565971 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.565954    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.566065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.566050    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.667936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.667898    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.668159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.667974    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.668309 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.668280    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb69q\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-kube-api-access-zb69q\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.668459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.668398    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.669139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.669082    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-plugins-config-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.669139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.669095    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-routing-overlay-volume\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.671358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.671329    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-provider-credentials\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.721279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.713331    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb69q\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-kube-api-access-zb69q\") pod \"payload-processing-external-model-5d5f4bbcf7-jtrbd\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") " pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:16.792283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:16.789552    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:17.068693 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.068486    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx" podUID="c4f6cc13-e393-4464-8016-e633c728a5b5" containerName="payload-processing" containerID="cri-o://322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc" gracePeriod=25
Nov 03 00:50:17.288364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.275150    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"]
Nov 03 00:50:17.291441 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:50:17.291409    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373ace8c_529f_48a0_9042_c55c3bc06c1a.slice/crio-24b8dfe2c9467b92301be44b6d467463217fd1c0bc8edbbd64f0f6ca36c85a2e WatchSource:0}: Error finding container 24b8dfe2c9467b92301be44b6d467463217fd1c0bc8edbbd64f0f6ca36c85a2e: Status 404 returned error can't find the container with id 24b8dfe2c9467b92301be44b6d467463217fd1c0bc8edbbd64f0f6ca36c85a2e
Nov 03 00:50:17.328121 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.328091    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 03 00:50:17.328585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.328244    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 03 00:50:17.347695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.347671    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:17.390923 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.390886    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-plugins-config-volume\") pod \"c4f6cc13-e393-4464-8016-e633c728a5b5\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") "
Nov 03 00:50:17.391085 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.390939    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-routing-overlay-volume\") pod \"c4f6cc13-e393-4464-8016-e633c728a5b5\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") "
Nov 03 00:50:17.391156 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.391141    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-provider-credentials\") pod \"c4f6cc13-e393-4464-8016-e633c728a5b5\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") "
Nov 03 00:50:17.391354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.391240    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-kube-api-access-jlc2r\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-kube-api-access-jlc2r\") pod \"c4f6cc13-e393-4464-8016-e633c728a5b5\" (UID: \"c4f6cc13-e393-4464-8016-e633c728a5b5\") "
Nov 03 00:50:17.391805 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.391780    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-routing-overlay-volume" pod "c4f6cc13-e393-4464-8016-e633c728a5b5" (UID: "c4f6cc13-e393-4464-8016-e633c728a5b5"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:17.391914 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.391903    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:17.392235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.392213    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-plugins-config-volume" pod "c4f6cc13-e393-4464-8016-e633c728a5b5" (UID: "c4f6cc13-e393-4464-8016-e633c728a5b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:17.396197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.395675    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-kube-api-access-jlc2r" pod "c4f6cc13-e393-4464-8016-e633c728a5b5" (UID: "c4f6cc13-e393-4464-8016-e633c728a5b5"). InnerVolumeSpecName "kube-api-access-jlc2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:17.396399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.396378    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-provider-credentials" pod "c4f6cc13-e393-4464-8016-e633c728a5b5" (UID: "c4f6cc13-e393-4464-8016-e633c728a5b5"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:17.494592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.494552    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:17.494592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.494583    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlc2r\" (UniqueName: \"kubernetes.io/projected/c4f6cc13-e393-4464-8016-e633c728a5b5-kube-api-access-jlc2r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:17.494592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:17.494598    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4f6cc13-e393-4464-8016-e633c728a5b5-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:18.139370 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.139338    6000 generic.go:356] "Generic (PLEG): container finished" podID="c4f6cc13-e393-4464-8016-e633c728a5b5" containerID="322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc" exitCode=0
Nov 03 00:50:18.139584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.139395    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"
Nov 03 00:50:18.139584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.139429    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx" event={"ID":"c4f6cc13-e393-4464-8016-e633c728a5b5","Type":"ContainerDied","Data":"322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc"}
Nov 03 00:50:18.139584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.139490    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx" event={"ID":"c4f6cc13-e393-4464-8016-e633c728a5b5","Type":"ContainerDied","Data":"350d34370b3c58b8fb74ebe0f99f35e37adf32465c68169b4a10f361e15b6a77"}
Nov 03 00:50:18.139584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.139520    6000 scope.go:122] "RemoveContainer" containerID="322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc"
Nov 03 00:50:18.141742 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.141711    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd" event={"ID":"373ace8c-529f-48a0-9042-c55c3bc06c1a","Type":"ContainerStarted","Data":"62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b"}
Nov 03 00:50:18.141865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.141754    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd" event={"ID":"373ace8c-529f-48a0-9042-c55c3bc06c1a","Type":"ContainerStarted","Data":"24b8dfe2c9467b92301be44b6d467463217fd1c0bc8edbbd64f0f6ca36c85a2e"}
Nov 03 00:50:18.141865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.141851    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:18.151091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.151075    6000 scope.go:122] "RemoveContainer" containerID="322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc"
Nov 03 00:50:18.151393 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:50:18.151374    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc\": container with ID starting with 322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc not found: ID does not exist" containerID="322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc"
Nov 03 00:50:18.151454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.151400    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc"} err="failed to get container status \"322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc\": rpc error: code = NotFound desc = could not find container \"322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc\": container with ID starting with 322d0d405188f383d3904d9e81beecc9f040ae39f67a8fff1903190c0dc715dc not found: ID does not exist"
Nov 03 00:50:18.212983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.212915    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd" podStartSLOduration=2.212896091 podStartE2EDuration="2.212896091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:18.209169135 +0000 UTC m=+2655.764521163" watchObservedRunningTime="2026-11-03 00:50:18.212896091 +0000 UTC m=+2655.768248116"
Nov 03 00:50:18.247437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.247372    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"]
Nov 03 00:50:18.252740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.252714    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-2ngxx"]
Nov 03 00:50:18.569788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:18.569712    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4f6cc13-e393-4464-8016-e633c728a5b5" path="/var/lib/kubelet/pods/c4f6cc13-e393-4464-8016-e633c728a5b5/volumes"
Nov 03 00:50:19.328112 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:19.328069    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-zqd94 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:50:19.328411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:19.328157    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:50:19.375280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:19.370744    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:19.375280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:19.372472    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:19.404283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:19.404157    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 00:50:19.903582 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:50:19.903536    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:50:20.581880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.581847    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"]
Nov 03 00:50:20.582767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.582750    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c4f6cc13-e393-4464-8016-e633c728a5b5" containerName="payload-processing"
Nov 03 00:50:20.582822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.582769    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4f6cc13-e393-4464-8016-e633c728a5b5" containerName="payload-processing"
Nov 03 00:50:20.582921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.582911    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4f6cc13-e393-4464-8016-e633c728a5b5" containerName="payload-processing"
Nov 03 00:50:20.588654 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.588632    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.606243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.606213    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"]
Nov 03 00:50:20.611867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.611836    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"agc-cred-5a75d901e3d5\""
Nov 03 00:50:20.611867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.611836    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"openshift-service-ca.crt\""
Nov 03 00:50:20.612062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.611873    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"routing-overlay\""
Nov 03 00:50:20.612062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.611880    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"kube-root-ca.crt\""
Nov 03 00:50:20.612062 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.611918    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d87781\""
Nov 03 00:50:20.614193 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.614172    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"payload-processing-external-model-e2e--2422cafc-dockercfg-z6gfg\""
Nov 03 00:50:20.704282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.701225    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.704901 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.704872    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mlx9\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-kube-api-access-2mlx9\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.717505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.717463    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.717859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.717834    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.831182 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.831139    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.831437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.831246    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.831511 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.831463    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.831578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.831559    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mlx9\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-kube-api-access-2mlx9\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.834882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.834378    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.835721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.835696    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.839318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.839111    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.861950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.861913    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mlx9\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-kube-api-access-2mlx9\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:20.907372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:20.907318    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:21.273549 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:50:21.273509    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88744866_3eaf_44ee_bc4b_4e5504901b06.slice/crio-530f0ef4ce791c1b20cadc0bb67f2ed75d6567cd2a02aa850fafa43240adf46d WatchSource:0}: Error finding container 530f0ef4ce791c1b20cadc0bb67f2ed75d6567cd2a02aa850fafa43240adf46d: Status 404 returned error can't find the container with id 530f0ef4ce791c1b20cadc0bb67f2ed75d6567cd2a02aa850fafa43240adf46d
Nov 03 00:50:21.274162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.274137    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"]
Nov 03 00:50:21.413325 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.413275    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6" podUID="717bd066-0fd1-442a-b476-055cb39bdf30" containerName="payload-processing" containerID="cri-o://1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8" gracePeriod=25
Nov 03 00:50:21.671356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.671329    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:21.851024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.850994    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-provider-credentials\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-provider-credentials\") pod \"717bd066-0fd1-442a-b476-055cb39bdf30\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") "
Nov 03 00:50:21.851160 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.851040    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-plugins-config-volume\") pod \"717bd066-0fd1-442a-b476-055cb39bdf30\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") "
Nov 03 00:50:21.851160 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.851085    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-kube-api-access-t4f5l\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-kube-api-access-t4f5l\") pod \"717bd066-0fd1-442a-b476-055cb39bdf30\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") "
Nov 03 00:50:21.851160 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.851121    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-routing-overlay-volume\") pod \"717bd066-0fd1-442a-b476-055cb39bdf30\" (UID: \"717bd066-0fd1-442a-b476-055cb39bdf30\") "
Nov 03 00:50:21.851550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.851524    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-plugins-config-volume" pod "717bd066-0fd1-442a-b476-055cb39bdf30" (UID: "717bd066-0fd1-442a-b476-055cb39bdf30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:21.851688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.851565    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-routing-overlay-volume" pod "717bd066-0fd1-442a-b476-055cb39bdf30" (UID: "717bd066-0fd1-442a-b476-055cb39bdf30"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:21.851953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.851891    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:21.851953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.851914    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/717bd066-0fd1-442a-b476-055cb39bdf30-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:21.853549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.853529    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-provider-credentials" pod "717bd066-0fd1-442a-b476-055cb39bdf30" (UID: "717bd066-0fd1-442a-b476-055cb39bdf30"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:21.853646 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.853600    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-kube-api-access-t4f5l" pod "717bd066-0fd1-442a-b476-055cb39bdf30" (UID: "717bd066-0fd1-442a-b476-055cb39bdf30"). InnerVolumeSpecName "kube-api-access-t4f5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:21.953079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.953050    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:21.953079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:21.953075    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4f5l\" (UniqueName: \"kubernetes.io/projected/717bd066-0fd1-442a-b476-055cb39bdf30-kube-api-access-t4f5l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:22.178467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.178422    6000 generic.go:356] "Generic (PLEG): container finished" podID="717bd066-0fd1-442a-b476-055cb39bdf30" containerID="1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8" exitCode=0
Nov 03 00:50:22.178467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.178467    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"
Nov 03 00:50:22.178697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.178464    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6" event={"ID":"717bd066-0fd1-442a-b476-055cb39bdf30","Type":"ContainerDied","Data":"1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8"}
Nov 03 00:50:22.178697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.178533    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6" event={"ID":"717bd066-0fd1-442a-b476-055cb39bdf30","Type":"ContainerDied","Data":"6b856df9a4c9d6c52266a6f063628d3e4ff32579e86498b3d357a2cd5f2b6794"}
Nov 03 00:50:22.178697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.178562    6000 scope.go:122] "RemoveContainer" containerID="1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8"
Nov 03 00:50:22.181107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.181071    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6" event={"ID":"88744866-3eaf-44ee-bc4b-4e5504901b06","Type":"ContainerStarted","Data":"01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0"}
Nov 03 00:50:22.181293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.181131    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:22.181293 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.181155    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6" event={"ID":"88744866-3eaf-44ee-bc4b-4e5504901b06","Type":"ContainerStarted","Data":"530f0ef4ce791c1b20cadc0bb67f2ed75d6567cd2a02aa850fafa43240adf46d"}
Nov 03 00:50:22.189873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.189829    6000 scope.go:122] "RemoveContainer" containerID="1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8"
Nov 03 00:50:22.190140 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:50:22.190119    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8\": container with ID starting with 1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8 not found: ID does not exist" containerID="1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8"
Nov 03 00:50:22.190214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.190150    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8"} err="failed to get container status \"1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8\": rpc error: code = NotFound desc = could not find container \"1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8\": container with ID starting with 1e68f53ab3447fa1f2fc101b20514b59aedec8ed78ebcd1cfefc617f06d702b8 not found: ID does not exist"
Nov 03 00:50:22.229168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.229139    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"]
Nov 03 00:50:22.240558 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.240530    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5db79bd7f6-9b7f6"]
Nov 03 00:50:22.267221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.267169    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6" podStartSLOduration=2.267154981 podStartE2EDuration="2.267154981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:22.26540049 +0000 UTC m=+2659.820752517" watchObservedRunningTime="2026-11-03 00:50:22.267154981 +0000 UTC m=+2659.822507005"
Nov 03 00:50:22.571983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:22.571910    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="717bd066-0fd1-442a-b476-055cb39bdf30" path="/var/lib/kubelet/pods/717bd066-0fd1-442a-b476-055cb39bdf30/volumes"
Nov 03 00:50:26.081188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:26.081156    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:50:29.158600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:29.158563    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:50:29.476190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:29.476082    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"]
Nov 03 00:50:33.193798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:33.193767    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:50:34.477020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.476974    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn" podUID="804e2180-7acf-4231-af26-c3e6ff11ff76" containerName="payload-processing" containerID="cri-o://50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe" gracePeriod=25
Nov 03 00:50:34.776314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.776287    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:50:34.787834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.787808    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-kube-api-access-7mbt8\") pod \"804e2180-7acf-4231-af26-c3e6ff11ff76\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") "
Nov 03 00:50:34.787980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.787866    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-routing-overlay-volume\") pod \"804e2180-7acf-4231-af26-c3e6ff11ff76\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") "
Nov 03 00:50:34.788061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.788029    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-plugins-config-volume\") pod \"804e2180-7acf-4231-af26-c3e6ff11ff76\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") "
Nov 03 00:50:34.788119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.788084    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-provider-credentials\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-provider-credentials\") pod \"804e2180-7acf-4231-af26-c3e6ff11ff76\" (UID: \"804e2180-7acf-4231-af26-c3e6ff11ff76\") "
Nov 03 00:50:34.788373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.788342    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-routing-overlay-volume" pod "804e2180-7acf-4231-af26-c3e6ff11ff76" (UID: "804e2180-7acf-4231-af26-c3e6ff11ff76"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:34.788469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.788434    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-plugins-config-volume" pod "804e2180-7acf-4231-af26-c3e6ff11ff76" (UID: "804e2180-7acf-4231-af26-c3e6ff11ff76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:50:34.788662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.788610    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:34.788662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.788629    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/804e2180-7acf-4231-af26-c3e6ff11ff76-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:34.790239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.790209    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-kube-api-access-7mbt8" pod "804e2180-7acf-4231-af26-c3e6ff11ff76" (UID: "804e2180-7acf-4231-af26-c3e6ff11ff76"). InnerVolumeSpecName "kube-api-access-7mbt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:34.790407 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.790377    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-provider-credentials" pod "804e2180-7acf-4231-af26-c3e6ff11ff76" (UID: "804e2180-7acf-4231-af26-c3e6ff11ff76"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:50:34.890092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.890059    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:34.890092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:34.890086    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7mbt8\" (UniqueName: \"kubernetes.io/projected/804e2180-7acf-4231-af26-c3e6ff11ff76-kube-api-access-7mbt8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:50:35.377472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.377432    6000 generic.go:356] "Generic (PLEG): container finished" podID="804e2180-7acf-4231-af26-c3e6ff11ff76" containerID="50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe" exitCode=0
Nov 03 00:50:35.377641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.377486    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"
Nov 03 00:50:35.377641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.377512    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn" event={"ID":"804e2180-7acf-4231-af26-c3e6ff11ff76","Type":"ContainerDied","Data":"50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe"}
Nov 03 00:50:35.377641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.377570    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn" event={"ID":"804e2180-7acf-4231-af26-c3e6ff11ff76","Type":"ContainerDied","Data":"e4d6e1e889b20c694c240cdc1f1c1ca1990735a4c59c466520b6829d83a78c93"}
Nov 03 00:50:35.377641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.377594    6000 scope.go:122] "RemoveContainer" containerID="50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe"
Nov 03 00:50:35.389843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.389818    6000 scope.go:122] "RemoveContainer" containerID="50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe"
Nov 03 00:50:35.390127 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:50:35.390110    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe\": container with ID starting with 50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe not found: ID does not exist" containerID="50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe"
Nov 03 00:50:35.390181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.390135    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe"} err="failed to get container status \"50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe\": rpc error: code = NotFound desc = could not find container \"50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe\": container with ID starting with 50b029b5f7ecdf4e780276b2d4807c21c07e73c916a17c464af8834e78e1f0fe not found: ID does not exist"
Nov 03 00:50:35.428526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.428494    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"]
Nov 03 00:50:35.439101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:35.439069    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-76d5d774bc-r25fn"]
Nov 03 00:50:36.569746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:36.569714    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804e2180-7acf-4231-af26-c3e6ff11ff76" path="/var/lib/kubelet/pods/804e2180-7acf-4231-af26-c3e6ff11ff76/volumes"
Nov 03 00:50:49.898641 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:50:49.898603    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:50:51.022387 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.022346    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"]
Nov 03 00:50:51.024343 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.024314    6000 cpu_manager.go:414] "containerMap: removing container" podUID="717bd066-0fd1-442a-b476-055cb39bdf30" containerName="payload-processing"
Nov 03 00:50:51.024567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.024553    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="717bd066-0fd1-442a-b476-055cb39bdf30" containerName="payload-processing"
Nov 03 00:50:51.024796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.024784    6000 cpu_manager.go:414] "containerMap: removing container" podUID="804e2180-7acf-4231-af26-c3e6ff11ff76" containerName="payload-processing"
Nov 03 00:50:51.024897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.024887    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804e2180-7acf-4231-af26-c3e6ff11ff76" containerName="payload-processing"
Nov 03 00:50:51.025226 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.025209    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="804e2180-7acf-4231-af26-c3e6ff11ff76" containerName="payload-processing"
Nov 03 00:50:51.025384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.025372    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="717bd066-0fd1-442a-b476-055cb39bdf30" containerName="payload-processing"
Nov 03 00:50:51.039244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.038648    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"]
Nov 03 00:50:51.039244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.038754    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.172882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.172846    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.173105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.172936    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wzlw\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-kube-api-access-9wzlw\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.173105 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.173002    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.173232 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.173123    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.280332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.280040    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.280332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.280204    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.280607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.280438    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.280607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.280511    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wzlw\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-kube-api-access-9wzlw\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.284278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.281772    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.284278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.283718    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.287714 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.287689    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.311149 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.305559    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wzlw\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-kube-api-access-9wzlw\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.356440 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.356406    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:51.678514 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:50:51.678471    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9782e69_f89e_4dcf_951f_e054d7f67bca.slice/crio-5cff18a091d422894ce9c8cbde95b18c60c0663a9cadf2bc143c8cb14b732f3f WatchSource:0}: Error finding container 5cff18a091d422894ce9c8cbde95b18c60c0663a9cadf2bc143c8cb14b732f3f: Status 404 returned error can't find the container with id 5cff18a091d422894ce9c8cbde95b18c60c0663a9cadf2bc143c8cb14b732f3f
Nov 03 00:50:51.678790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:51.678767    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"]
Nov 03 00:50:52.512996 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:52.512951    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp" event={"ID":"a9782e69-f89e-4dcf-951f-e054d7f67bca","Type":"ContainerStarted","Data":"143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61"}
Nov 03 00:50:52.513518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:52.513015    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp" event={"ID":"a9782e69-f89e-4dcf-951f-e054d7f67bca","Type":"ContainerStarted","Data":"5cff18a091d422894ce9c8cbde95b18c60c0663a9cadf2bc143c8cb14b732f3f"}
Nov 03 00:50:52.513518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:52.513041    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:50:52.554584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:50:52.554522    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp" podStartSLOduration=2.554506058 podStartE2EDuration="2.554506058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:50:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:50:52.550356255 +0000 UTC m=+2690.105708278" watchObservedRunningTime="2026-11-03 00:50:52.554506058 +0000 UTC m=+2690.109858082"
Nov 03 00:51:03.525474 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:03.525447    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:51:03.597665 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:03.596897    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"]
Nov 03 00:51:08.609273 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:08.606961    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8" podUID="f4b88892-130c-47a9-9ac8-cf656e6095dc" containerName="payload-processing" containerID="cri-o://5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277" gracePeriod=25
Nov 03 00:51:09.244782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.244753    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:51:09.326339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.326211    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-routing-overlay-volume\") pod \"f4b88892-130c-47a9-9ac8-cf656e6095dc\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") "
Nov 03 00:51:09.326522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.326358    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-plugins-config-volume\") pod \"f4b88892-130c-47a9-9ac8-cf656e6095dc\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") "
Nov 03 00:51:09.326522 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.326401    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-provider-credentials\") pod \"f4b88892-130c-47a9-9ac8-cf656e6095dc\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") "
Nov 03 00:51:09.326657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.326522    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-kube-api-access-d6hd2\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-kube-api-access-d6hd2\") pod \"f4b88892-130c-47a9-9ac8-cf656e6095dc\" (UID: \"f4b88892-130c-47a9-9ac8-cf656e6095dc\") "
Nov 03 00:51:09.330642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.326768    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-routing-overlay-volume" pod "f4b88892-130c-47a9-9ac8-cf656e6095dc" (UID: "f4b88892-130c-47a9-9ac8-cf656e6095dc"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:09.330642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.327370    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:09.330642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.330052    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-provider-credentials" pod "f4b88892-130c-47a9-9ac8-cf656e6095dc" (UID: "f4b88892-130c-47a9-9ac8-cf656e6095dc"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:09.331846 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.331815    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-plugins-config-volume" pod "f4b88892-130c-47a9-9ac8-cf656e6095dc" (UID: "f4b88892-130c-47a9-9ac8-cf656e6095dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:09.332444 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.332418    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-kube-api-access-d6hd2" pod "f4b88892-130c-47a9-9ac8-cf656e6095dc" (UID: "f4b88892-130c-47a9-9ac8-cf656e6095dc"). InnerVolumeSpecName "kube-api-access-d6hd2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:09.428391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.428360    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4b88892-130c-47a9-9ac8-cf656e6095dc-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:09.428391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.428386    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:09.428391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.428398    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6hd2\" (UniqueName: \"kubernetes.io/projected/f4b88892-130c-47a9-9ac8-cf656e6095dc-kube-api-access-d6hd2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:09.775107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.775074    6000 generic.go:356] "Generic (PLEG): container finished" podID="f4b88892-130c-47a9-9ac8-cf656e6095dc" containerID="5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277" exitCode=0
Nov 03 00:51:09.775598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.775139    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"
Nov 03 00:51:09.775598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.775158    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8" event={"ID":"f4b88892-130c-47a9-9ac8-cf656e6095dc","Type":"ContainerDied","Data":"5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277"}
Nov 03 00:51:09.775598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.775219    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8" event={"ID":"f4b88892-130c-47a9-9ac8-cf656e6095dc","Type":"ContainerDied","Data":"1fefe66290f65fc0352d95508b5359970b87e90b43eddd46c4fb294b9c7cb792"}
Nov 03 00:51:09.775598 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.775242    6000 scope.go:122] "RemoveContainer" containerID="5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277"
Nov 03 00:51:09.787965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.787940    6000 scope.go:122] "RemoveContainer" containerID="5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277"
Nov 03 00:51:09.788361 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:09.788246    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277\": container with ID starting with 5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277 not found: ID does not exist" containerID="5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277"
Nov 03 00:51:09.788461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.788374    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277"} err="failed to get container status \"5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277\": rpc error: code = NotFound desc = could not find container \"5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277\": container with ID starting with 5ea3593b71108d5ff27510104be0c5b796ef0c04d841335a61e8ac9501a18277 not found: ID does not exist"
Nov 03 00:51:09.825588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.825554    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"]
Nov 03 00:51:09.829084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:09.829064    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-6c882ksh8"]
Nov 03 00:51:10.570243 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:10.570214    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4b88892-130c-47a9-9ac8-cf656e6095dc" path="/var/lib/kubelet/pods/f4b88892-130c-47a9-9ac8-cf656e6095dc/volumes"
Nov 03 00:51:13.634347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.634186    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"]
Nov 03 00:51:13.635761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.635714    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f4b88892-130c-47a9-9ac8-cf656e6095dc" containerName="payload-processing"
Nov 03 00:51:13.635987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.635958    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4b88892-130c-47a9-9ac8-cf656e6095dc" containerName="payload-processing"
Nov 03 00:51:13.636687 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.636667    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4b88892-130c-47a9-9ac8-cf656e6095dc" containerName="payload-processing"
Nov 03 00:51:13.644567 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.644544    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.665316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.665278    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"]
Nov 03 00:51:13.708432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.708390    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxwkf\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-kube-api-access-hxwkf\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.708636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.708454    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.708636 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.708614    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.708776 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.708713    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.809834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.809793    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.810037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.809935    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxwkf\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-kube-api-access-hxwkf\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.810037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.809993    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.810169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.810138    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.810839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.810740    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.810839 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.810810    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.812681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.812655    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.829641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.829616    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxwkf\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-kube-api-access-hxwkf\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:13.961642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:13.961562    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:14.271085 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:51:14.271052    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e9d3b8d_0050_4bd4_9240_8a74f03b2fe2.slice/crio-9cf6b4ad2a4f106c6afe82c7bc8ffa67045f7fd14d705c58b3c6b79243ac0875 WatchSource:0}: Error finding container 9cf6b4ad2a4f106c6afe82c7bc8ffa67045f7fd14d705c58b3c6b79243ac0875: Status 404 returned error can't find the container with id 9cf6b4ad2a4f106c6afe82c7bc8ffa67045f7fd14d705c58b3c6b79243ac0875
Nov 03 00:51:14.272131 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:14.272102    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"]
Nov 03 00:51:14.339715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:14.339686    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"]
Nov 03 00:51:14.343268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:14.343227    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"]
Nov 03 00:51:14.869882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:14.869848    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g" event={"ID":"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2","Type":"ContainerStarted","Data":"3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15"}
Nov 03 00:51:14.870328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:14.869900    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g" event={"ID":"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2","Type":"ContainerStarted","Data":"9cf6b4ad2a4f106c6afe82c7bc8ffa67045f7fd14d705c58b3c6b79243ac0875"}
Nov 03 00:51:14.870328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:14.870048    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:14.909976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:14.909927    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g" podStartSLOduration=1.909908098 podStartE2EDuration="1.909908098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:14.906875436 +0000 UTC m=+2712.462227461" watchObservedRunningTime="2026-11-03 00:51:14.909908098 +0000 UTC m=+2712.465260121"
Nov 03 00:51:19.344510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.344473    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp" podUID="a9782e69-f89e-4dcf-951f-e054d7f67bca" containerName="payload-processing" containerID="cri-o://143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61" gracePeriod=25
Nov 03 00:51:19.622626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.622600    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:51:19.626681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.626657    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-routing-overlay-volume\") pod \"a9782e69-f89e-4dcf-951f-e054d7f67bca\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") "
Nov 03 00:51:19.626757 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.626719    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-provider-credentials\") pod \"a9782e69-f89e-4dcf-951f-e054d7f67bca\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") "
Nov 03 00:51:19.626800 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.626783    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-kube-api-access-9wzlw\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-kube-api-access-9wzlw\") pod \"a9782e69-f89e-4dcf-951f-e054d7f67bca\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") "
Nov 03 00:51:19.626972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.626950    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-plugins-config-volume\") pod \"a9782e69-f89e-4dcf-951f-e054d7f67bca\" (UID: \"a9782e69-f89e-4dcf-951f-e054d7f67bca\") "
Nov 03 00:51:19.627083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.627058    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-routing-overlay-volume" pod "a9782e69-f89e-4dcf-951f-e054d7f67bca" (UID: "a9782e69-f89e-4dcf-951f-e054d7f67bca"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:19.627585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.627518    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-plugins-config-volume" pod "a9782e69-f89e-4dcf-951f-e054d7f67bca" (UID: "a9782e69-f89e-4dcf-951f-e054d7f67bca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:19.628026 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.628001    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:19.628133 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.628043    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/a9782e69-f89e-4dcf-951f-e054d7f67bca-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:19.629734 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.629706    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-provider-credentials" pod "a9782e69-f89e-4dcf-951f-e054d7f67bca" (UID: "a9782e69-f89e-4dcf-951f-e054d7f67bca"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:19.629843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.629752    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-kube-api-access-9wzlw" pod "a9782e69-f89e-4dcf-951f-e054d7f67bca" (UID: "a9782e69-f89e-4dcf-951f-e054d7f67bca"). InnerVolumeSpecName "kube-api-access-9wzlw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:19.729681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.729647    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:19.729681 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.729680    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wzlw\" (UniqueName: \"kubernetes.io/projected/a9782e69-f89e-4dcf-951f-e054d7f67bca-kube-api-access-9wzlw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:19.870677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.870630    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g" podUID="9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" containerName="payload-processing" containerID="cri-o://3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15" gracePeriod=25
Nov 03 00:51:19.899341 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:19.899307    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:51:19.939725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.939695    6000 generic.go:356] "Generic (PLEG): container finished" podID="a9782e69-f89e-4dcf-951f-e054d7f67bca" containerID="143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61" exitCode=0
Nov 03 00:51:19.939879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.939761    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"
Nov 03 00:51:19.939879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.939775    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp" event={"ID":"a9782e69-f89e-4dcf-951f-e054d7f67bca","Type":"ContainerDied","Data":"143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61"}
Nov 03 00:51:19.939965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.939882    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp" event={"ID":"a9782e69-f89e-4dcf-951f-e054d7f67bca","Type":"ContainerDied","Data":"5cff18a091d422894ce9c8cbde95b18c60c0663a9cadf2bc143c8cb14b732f3f"}
Nov 03 00:51:19.939965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.939910    6000 scope.go:122] "RemoveContainer" containerID="143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61"
Nov 03 00:51:19.983865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.983847    6000 scope.go:122] "RemoveContainer" containerID="143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61"
Nov 03 00:51:19.984227 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:19.984208    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61\": container with ID starting with 143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61 not found: ID does not exist" containerID="143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61"
Nov 03 00:51:19.984320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.984237    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61"} err="failed to get container status \"143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61\": rpc error: code = NotFound desc = could not find container \"143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61\": container with ID starting with 143e21b155fe7a4d5a9c024c6d113b88c64a2c2ffa94e6353b0910c70d079c61 not found: ID does not exist"
Nov 03 00:51:19.994585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:19.994560    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"]
Nov 03 00:51:20.001661 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.001636    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-59d5j5ddp"]
Nov 03 00:51:20.154542 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.154518    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:20.236579 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.236546    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-plugins-config-volume\") pod \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") "
Nov 03 00:51:20.236777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.236722    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-kube-api-access-hxwkf\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-kube-api-access-hxwkf\") pod \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") "
Nov 03 00:51:20.236777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.236749    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-provider-credentials\") pod \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") "
Nov 03 00:51:20.236777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.236777    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-routing-overlay-volume\") pod \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\" (UID: \"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2\") "
Nov 03 00:51:20.237010 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.236979    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-plugins-config-volume" pod "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" (UID: "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:20.237355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.237327    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-routing-overlay-volume" pod "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" (UID: "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:20.237644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.237615    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:20.237748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.237647    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:20.239910 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.239879    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-kube-api-access-hxwkf" pod "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" (UID: "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2"). InnerVolumeSpecName "kube-api-access-hxwkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:20.240058 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.240037    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-provider-credentials" pod "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" (UID: "9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:20.339498 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.339465    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxwkf\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-kube-api-access-hxwkf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:20.339498 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.339495    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:20.569135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.569062    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9782e69-f89e-4dcf-951f-e054d7f67bca" path="/var/lib/kubelet/pods/a9782e69-f89e-4dcf-951f-e054d7f67bca/volumes"
Nov 03 00:51:20.948616 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.948584    6000 generic.go:356] "Generic (PLEG): container finished" podID="9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" containerID="3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15" exitCode=0
Nov 03 00:51:20.948823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.948632    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"
Nov 03 00:51:20.948823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.948665    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g" event={"ID":"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2","Type":"ContainerDied","Data":"3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15"}
Nov 03 00:51:20.948823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.948712    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g" event={"ID":"9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2","Type":"ContainerDied","Data":"9cf6b4ad2a4f106c6afe82c7bc8ffa67045f7fd14d705c58b3c6b79243ac0875"}
Nov 03 00:51:20.948823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.948733    6000 scope.go:122] "RemoveContainer" containerID="3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15"
Nov 03 00:51:20.962315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.962296    6000 scope.go:122] "RemoveContainer" containerID="3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15"
Nov 03 00:51:20.962664 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:20.962643    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15\": container with ID starting with 3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15 not found: ID does not exist" containerID="3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15"
Nov 03 00:51:20.962760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:20.962671    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15"} err="failed to get container status \"3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15\": rpc error: code = NotFound desc = could not find container \"3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15\": container with ID starting with 3a660de8cbc6f0b77a05a7a875f9e2e2a4009e8aafef94f624108425a144aa15 not found: ID does not exist"
Nov 03 00:51:21.001954 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:21.001880    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"]
Nov 03 00:51:21.041531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:21.041495    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-c974qs26g"]
Nov 03 00:51:22.571760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:22.571729    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" path="/var/lib/kubelet/pods/9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2/volumes"
Nov 03 00:51:29.373410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.373374    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"]
Nov 03 00:51:29.374224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.374209    6000 cpu_manager.go:414] "containerMap: removing container" podUID="a9782e69-f89e-4dcf-951f-e054d7f67bca" containerName="payload-processing"
Nov 03 00:51:29.374298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.374226    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9782e69-f89e-4dcf-951f-e054d7f67bca" containerName="payload-processing"
Nov 03 00:51:29.374298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.374279    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" containerName="payload-processing"
Nov 03 00:51:29.374298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.374286    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" containerName="payload-processing"
Nov 03 00:51:29.374419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.374408    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9782e69-f89e-4dcf-951f-e054d7f67bca" containerName="payload-processing"
Nov 03 00:51:29.374460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.374428    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e9d3b8d-0050-4bd4-9240-8a74f03b2fe2" containerName="payload-processing"
Nov 03 00:51:29.378178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.378158    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.395974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.395949    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"]
Nov 03 00:51:29.402532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.402507    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-c201f4488a8a\""
Nov 03 00:51:29.516241 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.516208    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7sq5\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-kube-api-access-l7sq5\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.516449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.516338    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-plugins-config-volume\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.516547 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.516523    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-routing-overlay-volume\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.516638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.516625    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-provider-credentials\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.619281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.618678    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-plugins-config-volume\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.619281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.618965    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-routing-overlay-volume\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.619281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.619136    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-provider-credentials\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.619613 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.619304    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7sq5\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-kube-api-access-l7sq5\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.620658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.620617    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-routing-overlay-volume\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.622391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.622366    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-plugins-config-volume\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.624850 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.624791    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-provider-credentials\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.637950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.637921    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7sq5\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-kube-api-access-l7sq5\") pod \"payload-processing-external-model-bc94678d4-jrb5l\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") " pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.693948 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.693921    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:29.997773 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:51:29.997745    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72612368_3f27_4241_ad42_3dfbf5a98605.slice/crio-1c742894a9f71a6a66cacdb22a0f8f4bddd726da043661c0a9d279c94661754d WatchSource:0}: Error finding container 1c742894a9f71a6a66cacdb22a0f8f4bddd726da043661c0a9d279c94661754d: Status 404 returned error can't find the container with id 1c742894a9f71a6a66cacdb22a0f8f4bddd726da043661c0a9d279c94661754d
Nov 03 00:51:29.998383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:29.998354    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"]
Nov 03 00:51:30.023703 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:30.023675    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l" event={"ID":"72612368-3f27-4241-ad42-3dfbf5a98605","Type":"ContainerStarted","Data":"1c742894a9f71a6a66cacdb22a0f8f4bddd726da043661c0a9d279c94661754d"}
Nov 03 00:51:31.033166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:31.033117    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l" event={"ID":"72612368-3f27-4241-ad42-3dfbf5a98605","Type":"ContainerStarted","Data":"f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76"}
Nov 03 00:51:31.033597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:31.033195    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:31.076806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:31.076752    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l" podStartSLOduration=2.076737248 podStartE2EDuration="2.076737248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:31.073115648 +0000 UTC m=+2728.628467672" watchObservedRunningTime="2026-11-03 00:51:31.076737248 +0000 UTC m=+2728.632089271"
Nov 03 00:51:32.753683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.753645    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"]
Nov 03 00:51:32.759559 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.759536    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.776071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.776047    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"]
Nov 03 00:51:32.883678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.883643    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.883867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.883754    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.883867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.883825    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwjl2\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-kube-api-access-dwjl2\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.883978 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.883914    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.985505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.985474    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.985726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.985562    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.985726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.985650    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwjl2\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-kube-api-access-dwjl2\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.985875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.985749    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.986317 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.986250    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.986428 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.986323    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:32.988520 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:32.988499    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:33.004482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:33.004413    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwjl2\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-kube-api-access-dwjl2\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:33.077113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:33.077074    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:33.570622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:33.570585    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"]
Nov 03 00:51:34.061173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.061141    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" event={"ID":"229052ea-7058-4bd7-9af1-78a5e941ae69","Type":"ContainerStarted","Data":"f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08"}
Nov 03 00:51:34.061631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.061196    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" event={"ID":"229052ea-7058-4bd7-9af1-78a5e941ae69","Type":"ContainerStarted","Data":"aa3ee00ded3d9bbfb7a94d203fd2e02e7a2a2eea372e91b01a8453493f55b116"}
Nov 03 00:51:34.061631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.061225    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:34.101488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.101431    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" podStartSLOduration=2.101416157 podStartE2EDuration="2.101416157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:34.09822497 +0000 UTC m=+2731.653576999" watchObservedRunningTime="2026-11-03 00:51:34.101416157 +0000 UTC m=+2731.656768180"
Nov 03 00:51:34.242184 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:34.242142    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-955dc3/agc-cred-35649651d663: secret "agc-cred-35649651d663" not found
Nov 03 00:51:34.242426 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:34.242212    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s: secret "agc-cred-35649651d663" not found
Nov 03 00:51:34.242426 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:34.242306    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials podName:229052ea-7058-4bd7-9af1-78a5e941ae69 nodeName:}" failed. No retries permitted until 2026-11-03 00:51:34.742289031 +0000 UTC m=+2732.297641046 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69") : secret "agc-cred-35649651d663" not found
Nov 03 00:51:34.533692 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.533658    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"]
Nov 03 00:51:34.571386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.571355    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"]
Nov 03 00:51:34.576663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.576642    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.586555 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.586528    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"]
Nov 03 00:51:34.764814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.764765    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddp5n\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-kube-api-access-ddp5n\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.765019 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.764850    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.765095 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.765042    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.765151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.765097    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.765455 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:34.765396    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-955dc3/agc-cred-35649651d663: secret "agc-cred-35649651d663" not found
Nov 03 00:51:34.765604 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:34.765592    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s: secret "agc-cred-35649651d663" not found
Nov 03 00:51:34.765728 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:34.765718    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials podName:229052ea-7058-4bd7-9af1-78a5e941ae69 nodeName:}" failed. No retries permitted until 2026-11-03 00:51:35.765698852 +0000 UTC m=+2733.321050869 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69") : secret "agc-cred-35649651d663" not found
Nov 03 00:51:34.866510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.866470    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.866703 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.866537    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.866760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.866732    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddp5n\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-kube-api-access-ddp5n\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.866814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.866792    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.867331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.867311    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.867847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.867815    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.869913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.869886    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.885707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.885680    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddp5n\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-kube-api-access-ddp5n\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:34.896536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:34.896517    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:35.280475 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:35.280396    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"]
Nov 03 00:51:35.372885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:35.372850    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"]
Nov 03 00:51:35.377369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:35.377344    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:35.800616 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:35.800585    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-955dc3/agc-cred-35649651d663: secret "agc-cred-35649651d663" not found
Nov 03 00:51:35.800616 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:35.800614    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s: secret "agc-cred-35649651d663" not found
Nov 03 00:51:35.800914 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:35.800680    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials podName:229052ea-7058-4bd7-9af1-78a5e941ae69 nodeName:}" failed. No retries permitted until 2026-11-03 00:51:37.800661187 +0000 UTC m=+2735.356013210 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69") : secret "agc-cred-35649651d663" not found
Nov 03 00:51:36.078127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:36.078024    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs" event={"ID":"4246cb36-40e4-4df5-acf0-189a7814a151","Type":"ContainerStarted","Data":"863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25"}
Nov 03 00:51:36.078127 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:36.078081    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs" event={"ID":"4246cb36-40e4-4df5-acf0-189a7814a151","Type":"ContainerStarted","Data":"418af550a1f04c6b7acc55fa88c5f6563b03db3f2aab78fdc23d7fd2ea53a902"}
Nov 03 00:51:36.078362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:36.078125    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:36.153733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:36.153683    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs" podStartSLOduration=2.153669027 podStartE2EDuration="2.153669027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:36.149578332 +0000 UTC m=+2733.704930358" watchObservedRunningTime="2026-11-03 00:51:36.153669027 +0000 UTC m=+2733.709021051"
Nov 03 00:51:37.834805 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:37.834765    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-955dc3/agc-cred-35649651d663: secret "agc-cred-35649651d663" not found
Nov 03 00:51:37.834805 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:37.834797    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s: secret "agc-cred-35649651d663" not found
Nov 03 00:51:37.835361 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:37.834884    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials podName:229052ea-7058-4bd7-9af1-78a5e941ae69 nodeName:}" failed. No retries permitted until 2026-11-03 00:51:41.834856062 +0000 UTC m=+2739.390208084 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69") : secret "agc-cred-35649651d663" not found
Nov 03 00:51:40.373713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.373676    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l" podUID="72612368-3f27-4241-ad42-3dfbf5a98605" containerName="payload-processing" containerID="cri-o://f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76" gracePeriod=25
Nov 03 00:51:40.740518 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.740433    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"]
Nov 03 00:51:40.747372 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.747350    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:40.764368 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.764336    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"]
Nov 03 00:51:40.770373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.769369    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"routing-overlay\""
Nov 03 00:51:40.770373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.770082    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d9e067\""
Nov 03 00:51:40.770373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.770146    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"kube-root-ca.crt\""
Nov 03 00:51:40.770373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.770225    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"agc-cred-ee02c64b9a9c\""
Nov 03 00:51:40.770373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.770298    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"openshift-service-ca.crt\""
Nov 03 00:51:40.776158 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.776131    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"payload-processing-external-model-e2e--b567d708-dockercfg-dv8tz\""
Nov 03 00:51:40.813446 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.813412    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:40.819825 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.819797    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:40.820124 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.820099    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4fpj\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-kube-api-access-p4fpj\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:40.820176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.820159    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:40.820305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:40.820282    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.018543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.004473    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-kube-api-access-l7sq5\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-kube-api-access-l7sq5\") pod \"72612368-3f27-4241-ad42-3dfbf5a98605\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") "
Nov 03 00:51:41.041926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.041891    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-plugins-config-volume\") pod \"72612368-3f27-4241-ad42-3dfbf5a98605\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") "
Nov 03 00:51:41.042470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.042446    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-provider-credentials\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-provider-credentials\") pod \"72612368-3f27-4241-ad42-3dfbf5a98605\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") "
Nov 03 00:51:41.042951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.042930    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-routing-overlay-volume\") pod \"72612368-3f27-4241-ad42-3dfbf5a98605\" (UID: \"72612368-3f27-4241-ad42-3dfbf5a98605\") "
Nov 03 00:51:41.043918 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.043889    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4fpj\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-kube-api-access-p4fpj\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.044108 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.044083    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.044270 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.044240    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.044602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.044587    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.046237 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.046217    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.046437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.023052    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-kube-api-access-l7sq5" pod "72612368-3f27-4241-ad42-3dfbf5a98605" (UID: "72612368-3f27-4241-ad42-3dfbf5a98605"). InnerVolumeSpecName "kube-api-access-l7sq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:41.047015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.046986    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-plugins-config-volume" pod "72612368-3f27-4241-ad42-3dfbf5a98605" (UID: "72612368-3f27-4241-ad42-3dfbf5a98605"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:41.048076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.048058    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.065277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.060046    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-routing-overlay-volume" pod "72612368-3f27-4241-ad42-3dfbf5a98605" (UID: "72612368-3f27-4241-ad42-3dfbf5a98605"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:41.065277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.062064    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-provider-credentials" pod "72612368-3f27-4241-ad42-3dfbf5a98605" (UID: "72612368-3f27-4241-ad42-3dfbf5a98605"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:41.065277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.064857    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.086774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.069940    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4fpj\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-kube-api-access-p4fpj\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.086774 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.080358    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" podUID="229052ea-7058-4bd7-9af1-78a5e941ae69" containerName="payload-processing" containerID="cri-o://f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08" gracePeriod=25
Nov 03 00:51:41.112395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.112361    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:41.122541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.122506    6000 generic.go:356] "Generic (PLEG): container finished" podID="72612368-3f27-4241-ad42-3dfbf5a98605" containerID="f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76" exitCode=0
Nov 03 00:51:41.122685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.122573    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l" event={"ID":"72612368-3f27-4241-ad42-3dfbf5a98605","Type":"ContainerDied","Data":"f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76"}
Nov 03 00:51:41.122685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.122638    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"
Nov 03 00:51:41.122685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.122656    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l" event={"ID":"72612368-3f27-4241-ad42-3dfbf5a98605","Type":"ContainerDied","Data":"1c742894a9f71a6a66cacdb22a0f8f4bddd726da043661c0a9d279c94661754d"}
Nov 03 00:51:41.122685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.122679    6000 scope.go:122] "RemoveContainer" containerID="f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76"
Nov 03 00:51:41.147342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.147295    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.147482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.147354    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7sq5\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-kube-api-access-l7sq5\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.147482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.147373    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/72612368-3f27-4241-ad42-3dfbf5a98605-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.147482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.147392    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/72612368-3f27-4241-ad42-3dfbf5a98605-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.191042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.189849    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"]
Nov 03 00:51:41.203496 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.202640    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-bc94678d4-jrb5l"]
Nov 03 00:51:41.217618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.217072    6000 scope.go:122] "RemoveContainer" containerID="f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76"
Nov 03 00:51:41.220280 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:41.219898    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76\": container with ID starting with f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76 not found: ID does not exist" containerID="f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76"
Nov 03 00:51:41.220280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.219943    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76"} err="failed to get container status \"f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76\": rpc error: code = NotFound desc = could not find container \"f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76\": container with ID starting with f938e609daa50b57221c7d6450786d67a0c0349bc872038be25f26953b727c76 not found: ID does not exist"
Nov 03 00:51:41.439004 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.438975    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"]
Nov 03 00:51:41.442770 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:51:41.442735    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977794cf_6dd1_43c8_a000_0f885470cae7.slice/crio-789cefdd87af798f267a11e52918bb8251b11a48c2e85d8998b82c5bd4242fe2 WatchSource:0}: Error finding container 789cefdd87af798f267a11e52918bb8251b11a48c2e85d8998b82c5bd4242fe2: Status 404 returned error can't find the container with id 789cefdd87af798f267a11e52918bb8251b11a48c2e85d8998b82c5bd4242fe2
Nov 03 00:51:41.476361 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.476337    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:41.555155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.555128    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-routing-overlay-volume\") pod \"229052ea-7058-4bd7-9af1-78a5e941ae69\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") "
Nov 03 00:51:41.555298 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.555279    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-kube-api-access-dwjl2\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-kube-api-access-dwjl2\") pod \"229052ea-7058-4bd7-9af1-78a5e941ae69\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") "
Nov 03 00:51:41.555419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.555402    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials\") pod \"229052ea-7058-4bd7-9af1-78a5e941ae69\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") "
Nov 03 00:51:41.555531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.555512    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-plugins-config-volume\") pod \"229052ea-7058-4bd7-9af1-78a5e941ae69\" (UID: \"229052ea-7058-4bd7-9af1-78a5e941ae69\") "
Nov 03 00:51:41.555595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.555522    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-routing-overlay-volume" pod "229052ea-7058-4bd7-9af1-78a5e941ae69" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:41.556155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.556041    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-plugins-config-volume" pod "229052ea-7058-4bd7-9af1-78a5e941ae69" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:41.556319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.556250    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.561316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.558494    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-kube-api-access-dwjl2" pod "229052ea-7058-4bd7-9af1-78a5e941ae69" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69"). InnerVolumeSpecName "kube-api-access-dwjl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:41.561316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.559715    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials" pod "229052ea-7058-4bd7-9af1-78a5e941ae69" (UID: "229052ea-7058-4bd7-9af1-78a5e941ae69"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:41.658189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.658152    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwjl2\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-kube-api-access-dwjl2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.658189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.658190    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/229052ea-7058-4bd7-9af1-78a5e941ae69-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.658453 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.658206    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/229052ea-7058-4bd7-9af1-78a5e941ae69-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:41.751733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.751696    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"]
Nov 03 00:51:41.752601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.752580    6000 cpu_manager.go:414] "containerMap: removing container" podUID="72612368-3f27-4241-ad42-3dfbf5a98605" containerName="payload-processing"
Nov 03 00:51:41.752601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.752599    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72612368-3f27-4241-ad42-3dfbf5a98605" containerName="payload-processing"
Nov 03 00:51:41.752775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.752633    6000 cpu_manager.go:414] "containerMap: removing container" podUID="229052ea-7058-4bd7-9af1-78a5e941ae69" containerName="payload-processing"
Nov 03 00:51:41.752775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.752639    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="229052ea-7058-4bd7-9af1-78a5e941ae69" containerName="payload-processing"
Nov 03 00:51:41.752849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.752784    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="72612368-3f27-4241-ad42-3dfbf5a98605" containerName="payload-processing"
Nov 03 00:51:41.752849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.752801    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="229052ea-7058-4bd7-9af1-78a5e941ae69" containerName="payload-processing"
Nov 03 00:51:41.756952 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.756934    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:41.770970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.770913    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"routing-overlay\""
Nov 03 00:51:41.770970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.770913    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-plugins-e2e-extmod-b-fdacdb\""
Nov 03 00:51:41.771198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.771165    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"openshift-service-ca.crt\""
Nov 03 00:51:41.771305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.771202    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"payload-processing-external-model-e2e--2e23f326-dockercfg-5gcjj\""
Nov 03 00:51:41.771305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.771213    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-eeb93388eae4\""
Nov 03 00:51:41.771432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.771367    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"kube-root-ca.crt\""
Nov 03 00:51:41.775540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.775516    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"]
Nov 03 00:51:41.879443 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.879402    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:41.880398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.880370    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:41.880862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.880751    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p527w\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-kube-api-access-p527w\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:41.881031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.881013    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.004346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.986032    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p527w\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-kube-api-access-p527w\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.004346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.986095    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.004346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.986337    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.004346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.986552    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.004346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.987959    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.004346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:41.995106    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.021424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.021232    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.033984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.033903    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p527w\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-kube-api-access-p527w\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.073573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.073524    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:42.136404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.136296    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm" event={"ID":"977794cf-6dd1-43c8-a000-0f885470cae7","Type":"ContainerStarted","Data":"9a153d085b0e7b1454274e1c3c512dd1f1ceb92c9a13f7a0fe1c1282ca4ee0d7"}
Nov 03 00:51:42.136705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.136674    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:42.136834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.136717    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm" event={"ID":"977794cf-6dd1-43c8-a000-0f885470cae7","Type":"ContainerStarted","Data":"789cefdd87af798f267a11e52918bb8251b11a48c2e85d8998b82c5bd4242fe2"}
Nov 03 00:51:42.139557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.139516    6000 generic.go:356] "Generic (PLEG): container finished" podID="229052ea-7058-4bd7-9af1-78a5e941ae69" containerID="f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08" exitCode=0
Nov 03 00:51:42.139730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.139606    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" event={"ID":"229052ea-7058-4bd7-9af1-78a5e941ae69","Type":"ContainerDied","Data":"f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08"}
Nov 03 00:51:42.139730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.139648    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s" event={"ID":"229052ea-7058-4bd7-9af1-78a5e941ae69","Type":"ContainerDied","Data":"aa3ee00ded3d9bbfb7a94d203fd2e02e7a2a2eea372e91b01a8453493f55b116"}
Nov 03 00:51:42.139730 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.139677    6000 scope.go:122] "RemoveContainer" containerID="f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08"
Nov 03 00:51:42.139983 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.139790    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"
Nov 03 00:51:42.162668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.162638    6000 scope.go:122] "RemoveContainer" containerID="f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08"
Nov 03 00:51:42.163081 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:42.163053    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08\": container with ID starting with f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08 not found: ID does not exist" containerID="f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08"
Nov 03 00:51:42.163194 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.163103    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08"} err="failed to get container status \"f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08\": rpc error: code = NotFound desc = could not find container \"f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08\": container with ID starting with f3bfadbee9592ca35cb163e5e434a999a212bd8722ab5221844372f016f1fe08 not found: ID does not exist"
Nov 03 00:51:42.223704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.223651    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm" podStartSLOduration=2.223636454 podStartE2EDuration="2.223636454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:42.183499271 +0000 UTC m=+2739.738851308" watchObservedRunningTime="2026-11-03 00:51:42.223636454 +0000 UTC m=+2739.778988478"
Nov 03 00:51:42.231033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.228341    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"]
Nov 03 00:51:42.234470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.234436    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-5fbbzp98s"]
Nov 03 00:51:42.384561 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:51:42.384535    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7dcb737_431f_433b_a63b_1adaac0f081e.slice/crio-16a6c821975ada328cd506dc3321e71176f5a164002e1ce4df658672764a0e4f WatchSource:0}: Error finding container 16a6c821975ada328cd506dc3321e71176f5a164002e1ce4df658672764a0e4f: Status 404 returned error can't find the container with id 16a6c821975ada328cd506dc3321e71176f5a164002e1ce4df658672764a0e4f
Nov 03 00:51:42.385911 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.385891    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"]
Nov 03 00:51:42.570032 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.570001    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="229052ea-7058-4bd7-9af1-78a5e941ae69" path="/var/lib/kubelet/pods/229052ea-7058-4bd7-9af1-78a5e941ae69/volumes"
Nov 03 00:51:42.570461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:42.570446    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72612368-3f27-4241-ad42-3dfbf5a98605" path="/var/lib/kubelet/pods/72612368-3f27-4241-ad42-3dfbf5a98605/volumes"
Nov 03 00:51:43.098430 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:43.098393    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:51:43.149882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:43.149840    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2" event={"ID":"c7dcb737-431f-433b-a63b-1adaac0f081e","Type":"ContainerStarted","Data":"be8d286be325522b119c6abab410ce6b675745c05834bc8313a5bda4cf28b60f"}
Nov 03 00:51:43.150063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:43.149910    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2" event={"ID":"c7dcb737-431f-433b-a63b-1adaac0f081e","Type":"ContainerStarted","Data":"16a6c821975ada328cd506dc3321e71176f5a164002e1ce4df658672764a0e4f"}
Nov 03 00:51:43.150063 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:43.150061    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:51:43.193023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:43.192971    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2" podStartSLOduration=2.192958078 podStartE2EDuration="2.192958078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:51:43.190588429 +0000 UTC m=+2740.745940456" watchObservedRunningTime="2026-11-03 00:51:43.192958078 +0000 UTC m=+2740.748310102"
Nov 03 00:51:46.785402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:46.780208    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"]
Nov 03 00:51:46.898286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:46.896536    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"]
Nov 03 00:51:46.982894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:46.982661    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:49.897690 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:49.897654    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:51:51.780653 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:51.780611    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn" podUID="66695227-d8ac-4cae-b4b8-b334414a832a" containerName="payload-processing" containerID="cri-o://6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86" gracePeriod=25
Nov 03 00:51:51.897297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:51.897241    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs" podUID="4246cb36-40e4-4df5-acf0-189a7814a151" containerName="payload-processing" containerID="cri-o://863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25" gracePeriod=25
Nov 03 00:51:52.081212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.081185    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:51:52.174507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.174472    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-plugins-config-volume\") pod \"66695227-d8ac-4cae-b4b8-b334414a832a\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") "
Nov 03 00:51:52.174683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.174654    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-routing-overlay-volume\") pod \"66695227-d8ac-4cae-b4b8-b334414a832a\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") "
Nov 03 00:51:52.174770 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.174739    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-provider-credentials\") pod \"66695227-d8ac-4cae-b4b8-b334414a832a\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") "
Nov 03 00:51:52.174865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.174846    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-kube-api-access-8hbdm\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-kube-api-access-8hbdm\") pod \"66695227-d8ac-4cae-b4b8-b334414a832a\" (UID: \"66695227-d8ac-4cae-b4b8-b334414a832a\") "
Nov 03 00:51:52.175046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.174888    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-plugins-config-volume" pod "66695227-d8ac-4cae-b4b8-b334414a832a" (UID: "66695227-d8ac-4cae-b4b8-b334414a832a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:52.175114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.175064    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-routing-overlay-volume" pod "66695227-d8ac-4cae-b4b8-b334414a832a" (UID: "66695227-d8ac-4cae-b4b8-b334414a832a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:52.175476 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.175459    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.175544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.175480    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/66695227-d8ac-4cae-b4b8-b334414a832a-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.177080 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.177052    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-provider-credentials" pod "66695227-d8ac-4cae-b4b8-b334414a832a" (UID: "66695227-d8ac-4cae-b4b8-b334414a832a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:52.177238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.177213    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-kube-api-access-8hbdm" pod "66695227-d8ac-4cae-b4b8-b334414a832a" (UID: "66695227-d8ac-4cae-b4b8-b334414a832a"). InnerVolumeSpecName "kube-api-access-8hbdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:52.203988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.203966    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:52.276326 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.276247    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-provider-credentials\") pod \"4246cb36-40e4-4df5-acf0-189a7814a151\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") "
Nov 03 00:51:52.276529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.276459    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-routing-overlay-volume\") pod \"4246cb36-40e4-4df5-acf0-189a7814a151\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") "
Nov 03 00:51:52.276529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.276495    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-plugins-config-volume\") pod \"4246cb36-40e4-4df5-acf0-189a7814a151\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") "
Nov 03 00:51:52.276529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.276526    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-kube-api-access-ddp5n\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-kube-api-access-ddp5n\") pod \"4246cb36-40e4-4df5-acf0-189a7814a151\" (UID: \"4246cb36-40e4-4df5-acf0-189a7814a151\") "
Nov 03 00:51:52.276947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.276923    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-routing-overlay-volume" pod "4246cb36-40e4-4df5-acf0-189a7814a151" (UID: "4246cb36-40e4-4df5-acf0-189a7814a151"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:52.277046 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.276959    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-plugins-config-volume" pod "4246cb36-40e4-4df5-acf0-189a7814a151" (UID: "4246cb36-40e4-4df5-acf0-189a7814a151"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:51:52.277493 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.277468    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.277621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.277497    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4246cb36-40e4-4df5-acf0-189a7814a151-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.277621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.277514    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.277621 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.277529    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hbdm\" (UniqueName: \"kubernetes.io/projected/66695227-d8ac-4cae-b4b8-b334414a832a-kube-api-access-8hbdm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.279044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.279013    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-provider-credentials" pod "4246cb36-40e4-4df5-acf0-189a7814a151" (UID: "4246cb36-40e4-4df5-acf0-189a7814a151"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:52.279044 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.279027    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-kube-api-access-ddp5n" pod "4246cb36-40e4-4df5-acf0-189a7814a151" (UID: "4246cb36-40e4-4df5-acf0-189a7814a151"). InnerVolumeSpecName "kube-api-access-ddp5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:51:52.336814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.336733    6000 generic.go:356] "Generic (PLEG): container finished" podID="4246cb36-40e4-4df5-acf0-189a7814a151" containerID="863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25" exitCode=0
Nov 03 00:51:52.336814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.336785    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"
Nov 03 00:51:52.337053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.336821    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs" event={"ID":"4246cb36-40e4-4df5-acf0-189a7814a151","Type":"ContainerDied","Data":"863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25"}
Nov 03 00:51:52.337053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.336881    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs" event={"ID":"4246cb36-40e4-4df5-acf0-189a7814a151","Type":"ContainerDied","Data":"418af550a1f04c6b7acc55fa88c5f6563b03db3f2aab78fdc23d7fd2ea53a902"}
Nov 03 00:51:52.337053 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.336907    6000 scope.go:122] "RemoveContainer" containerID="863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25"
Nov 03 00:51:52.338607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.338581    6000 generic.go:356] "Generic (PLEG): container finished" podID="66695227-d8ac-4cae-b4b8-b334414a832a" containerID="6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86" exitCode=0
Nov 03 00:51:52.338708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.338671    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn" event={"ID":"66695227-d8ac-4cae-b4b8-b334414a832a","Type":"ContainerDied","Data":"6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86"}
Nov 03 00:51:52.338760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.338715    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn" event={"ID":"66695227-d8ac-4cae-b4b8-b334414a832a","Type":"ContainerDied","Data":"948d1efa42e084ad6ab54df5981ad33c1d1db21963dcd7b3c4f1c1d4eb482720"}
Nov 03 00:51:52.338842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.338822    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"
Nov 03 00:51:52.348627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.348506    6000 scope.go:122] "RemoveContainer" containerID="863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25"
Nov 03 00:51:52.348837 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:52.348815    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25\": container with ID starting with 863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25 not found: ID does not exist" containerID="863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25"
Nov 03 00:51:52.348922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.348845    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25"} err="failed to get container status \"863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25\": rpc error: code = NotFound desc = could not find container \"863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25\": container with ID starting with 863cbab0f842b9644f1390380f557b36f2efe061b744694f01b7dffbd0ddbc25 not found: ID does not exist"
Nov 03 00:51:52.348922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.348862    6000 scope.go:122] "RemoveContainer" containerID="6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86"
Nov 03 00:51:52.358955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.358936    6000 scope.go:122] "RemoveContainer" containerID="6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86"
Nov 03 00:51:52.359219 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:51:52.359199    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86\": container with ID starting with 6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86 not found: ID does not exist" containerID="6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86"
Nov 03 00:51:52.359316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.359225    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86"} err="failed to get container status \"6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86\": rpc error: code = NotFound desc = could not find container \"6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86\": container with ID starting with 6fbbba2336f8fcc5d116773dbfd96310183b51da3a434cf77ab49e4161a45e86 not found: ID does not exist"
Nov 03 00:51:52.378953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.378886    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.378953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.378949    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ddp5n\" (UniqueName: \"kubernetes.io/projected/4246cb36-40e4-4df5-acf0-189a7814a151-kube-api-access-ddp5n\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:51:52.390269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.390211    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"]
Nov 03 00:51:52.394589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.394568    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-644f7w8tn"]
Nov 03 00:51:52.426615 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.426584    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"]
Nov 03 00:51:52.430310 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.430285    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6784h9ngs"]
Nov 03 00:51:52.571587 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.571548    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4246cb36-40e4-4df5-acf0-189a7814a151" path="/var/lib/kubelet/pods/4246cb36-40e4-4df5-acf0-189a7814a151/volumes"
Nov 03 00:51:52.572210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:52.572151    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66695227-d8ac-4cae-b4b8-b334414a832a" path="/var/lib/kubelet/pods/66695227-d8ac-4cae-b4b8-b334414a832a/volumes"
Nov 03 00:51:53.154717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:53.154690    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:51:54.166056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:51:54.166029    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:52:02.072092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.072056    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"]
Nov 03 00:52:02.072980 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.072963    6000 cpu_manager.go:414] "containerMap: removing container" podUID="4246cb36-40e4-4df5-acf0-189a7814a151" containerName="payload-processing"
Nov 03 00:52:02.073045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.072983    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4246cb36-40e4-4df5-acf0-189a7814a151" containerName="payload-processing"
Nov 03 00:52:02.073045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.073032    6000 cpu_manager.go:414] "containerMap: removing container" podUID="66695227-d8ac-4cae-b4b8-b334414a832a" containerName="payload-processing"
Nov 03 00:52:02.073045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.073039    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66695227-d8ac-4cae-b4b8-b334414a832a" containerName="payload-processing"
Nov 03 00:52:02.073188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.073169    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="4246cb36-40e4-4df5-acf0-189a7814a151" containerName="payload-processing"
Nov 03 00:52:02.073188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.073193    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="66695227-d8ac-4cae-b4b8-b334414a832a" containerName="payload-processing"
Nov 03 00:52:02.077179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.077157    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.092222 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.092197    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"agc-cred-824928b36b76\""
Nov 03 00:52:02.109186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.104017    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"kube-root-ca.crt\""
Nov 03 00:52:02.109186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.104017    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"openshift-service-ca.crt\""
Nov 03 00:52:02.109186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.104056    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-plugins-e2e-extmod-a-955dc3\""
Nov 03 00:52:02.109186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.104061    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"routing-overlay\""
Nov 03 00:52:02.109186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.104087    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-955dc3\"/\"payload-processing-external-model-e2e--70281560-dockercfg-q9cjv\""
Nov 03 00:52:02.109186 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.109109    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"]
Nov 03 00:52:02.263408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.263348    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.263637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.263595    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcms7\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-kube-api-access-xcms7\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.263759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.263744    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.263816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.263767    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.374280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.367014    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcms7\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-kube-api-access-xcms7\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.374280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.367163    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.374280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.367193    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.374280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.367371    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.374280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.368825    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.374280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.370863    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.397283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.384228    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.414959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.405361    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcms7\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-kube-api-access-xcms7\") pod \"payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") " pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.414959 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.407127    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:02.873758 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:52:02.873724    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf90eb3f_8fcf_4ed5_b57e_f6e62754ab5f.slice/crio-537857d5eddc588f371bbcbc58adb67c078c157206a73221bdd8055edc2717f1 WatchSource:0}: Error finding container 537857d5eddc588f371bbcbc58adb67c078c157206a73221bdd8055edc2717f1: Status 404 returned error can't find the container with id 537857d5eddc588f371bbcbc58adb67c078c157206a73221bdd8055edc2717f1
Nov 03 00:52:02.874751 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:02.874725    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"]
Nov 03 00:52:03.553552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:03.553515    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq" event={"ID":"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f","Type":"ContainerStarted","Data":"3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb"}
Nov 03 00:52:03.553949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:03.553577    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq" event={"ID":"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f","Type":"ContainerStarted","Data":"537857d5eddc588f371bbcbc58adb67c078c157206a73221bdd8055edc2717f1"}
Nov 03 00:52:03.553949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:03.553599    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:03.595139 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:03.595088    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq" podStartSLOduration=1.595073886 podStartE2EDuration="1.595073886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:03.592716539 +0000 UTC m=+2761.148068564" watchObservedRunningTime="2026-11-03 00:52:03.595073886 +0000 UTC m=+2761.150425909"
Nov 03 00:52:03.842755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:03.842677    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"]
Nov 03 00:52:05.383977 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.383943    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"]
Nov 03 00:52:05.389437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.389410    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.407533 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.407501    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"]
Nov 03 00:52:05.413869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.413841    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"openshift-service-ca.crt\""
Nov 03 00:52:05.413869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.413843    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"payload-processing-external-model-e2e--4be0dbee-dockercfg-bh6mg\""
Nov 03 00:52:05.414110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.413886    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"agc-cred-926d1a82a237\""
Nov 03 00:52:05.417400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.417337    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"payload-processing-external-model-plugins-e2e-extmod-b-909268\""
Nov 03 00:52:05.417400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.417361    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"routing-overlay\""
Nov 03 00:52:05.417400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.417399    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-909268\"/\"kube-root-ca.crt\""
Nov 03 00:52:05.557281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.556666    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.557281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.556920    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.557281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.557138    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.557624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.557501    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xjm5\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-kube-api-access-8xjm5\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.663652 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.663533    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.663934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.663898    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.664166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.664137    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xjm5\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-kube-api-access-8xjm5\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.664589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.664555    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.665789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.665751    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.669648 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.669628    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.670530 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.670510    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.695047 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.692899    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xjm5\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-kube-api-access-8xjm5\") pod \"payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") " pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:05.712278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:05.711215    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:06.057133 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:52:06.056961    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86eea21a_168f_4818_b94d_582ee02e4445.slice/crio-4f59730f930cae805ac2cf067c4f0223ac6e757be1f38e87a69a10a1e6b12704 WatchSource:0}: Error finding container 4f59730f930cae805ac2cf067c4f0223ac6e757be1f38e87a69a10a1e6b12704: Status 404 returned error can't find the container with id 4f59730f930cae805ac2cf067c4f0223ac6e757be1f38e87a69a10a1e6b12704
Nov 03 00:52:06.057593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:06.057558    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"]
Nov 03 00:52:06.791278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:06.781668    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz" event={"ID":"86eea21a-168f-4818-b94d-582ee02e4445","Type":"ContainerStarted","Data":"35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c"}
Nov 03 00:52:06.791278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:06.781737    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz" event={"ID":"86eea21a-168f-4818-b94d-582ee02e4445","Type":"ContainerStarted","Data":"4f59730f930cae805ac2cf067c4f0223ac6e757be1f38e87a69a10a1e6b12704"}
Nov 03 00:52:06.791278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:06.782966    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:07.998710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:07.998654    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz" podStartSLOduration=2.998638864 podStartE2EDuration="2.998638864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:06.830710111 +0000 UTC m=+2764.386062135" watchObservedRunningTime="2026-11-03 00:52:07.998638864 +0000 UTC m=+2765.553990891"
Nov 03 00:52:08.000088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.000065    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"]
Nov 03 00:52:08.005305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.005287    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.029907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.029877    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"agc-cred-f719de565e1c\""
Nov 03 00:52:08.029907 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.029896    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"kube-root-ca.crt\""
Nov 03 00:52:08.030090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.029877    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"routing-overlay\""
Nov 03 00:52:08.030090 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.029968    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"payload-processing-external-model-e2e--0fa13bfc-dockercfg-v78wj\""
Nov 03 00:52:08.034321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.034300    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"payload-processing-external-model-plugins-e2e-extmod-b-ac338d\""
Nov 03 00:52:08.034433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.034405    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"openshift-service-ca.crt\""
Nov 03 00:52:08.046079 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.046049    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"]
Nov 03 00:52:08.053210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.053177    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.053392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.053321    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.053392 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.053388    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sw6j\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-kube-api-access-7sw6j\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.053546 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.053499    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.266626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.253052    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sw6j\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-kube-api-access-7sw6j\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.266626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.261116    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.266626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.261420    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.266626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.261646    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.266626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.262576    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.274456 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.274364    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.281961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.281763    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.288564 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.287917    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sw6j\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-kube-api-access-7sw6j\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.322364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.322327    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.676133 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:52:08.676102    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64dd5195_46b0_4e2d_9960_7d6a6026aa04.slice/crio-f7c48ecce0204d5b2b834e4fbe8c3d371ffb8cee86c4fa839541e65a0c82a729 WatchSource:0}: Error finding container f7c48ecce0204d5b2b834e4fbe8c3d371ffb8cee86c4fa839541e65a0c82a729: Status 404 returned error can't find the container with id f7c48ecce0204d5b2b834e4fbe8c3d371ffb8cee86c4fa839541e65a0c82a729
Nov 03 00:52:08.676764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.676635    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"]
Nov 03 00:52:08.799958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.799903    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf" event={"ID":"64dd5195-46b0-4e2d-9960-7d6a6026aa04","Type":"ContainerStarted","Data":"8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889"}
Nov 03 00:52:08.800482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.799974    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf" event={"ID":"64dd5195-46b0-4e2d-9960-7d6a6026aa04","Type":"ContainerStarted","Data":"f7c48ecce0204d5b2b834e4fbe8c3d371ffb8cee86c4fa839541e65a0c82a729"}
Nov 03 00:52:08.800482 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.800007    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:08.838056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.837998    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf" podStartSLOduration=1.837983108 podStartE2EDuration="1.837983108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:08.836515556 +0000 UTC m=+2766.391867581" watchObservedRunningTime="2026-11-03 00:52:08.837983108 +0000 UTC m=+2766.393335131"
Nov 03 00:52:08.843399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:08.843361    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd" podUID="373ace8c-529f-48a0-9042-c55c3bc06c1a" containerName="payload-processing" containerID="cri-o://62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b" gracePeriod=25
Nov 03 00:52:09.185560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.185530    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:52:09.277964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.277930    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-routing-overlay-volume\") pod \"373ace8c-529f-48a0-9042-c55c3bc06c1a\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") "
Nov 03 00:52:09.278183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.278057    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-kube-api-access-zb69q\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-kube-api-access-zb69q\") pod \"373ace8c-529f-48a0-9042-c55c3bc06c1a\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") "
Nov 03 00:52:09.278183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.278137    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-plugins-config-volume\") pod \"373ace8c-529f-48a0-9042-c55c3bc06c1a\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") "
Nov 03 00:52:09.278381 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.278211    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-provider-credentials\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-provider-credentials\") pod \"373ace8c-529f-48a0-9042-c55c3bc06c1a\" (UID: \"373ace8c-529f-48a0-9042-c55c3bc06c1a\") "
Nov 03 00:52:09.279155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.279021    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-routing-overlay-volume" pod "373ace8c-529f-48a0-9042-c55c3bc06c1a" (UID: "373ace8c-529f-48a0-9042-c55c3bc06c1a"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:52:09.279523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.279488    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-plugins-config-volume" pod "373ace8c-529f-48a0-9042-c55c3bc06c1a" (UID: "373ace8c-529f-48a0-9042-c55c3bc06c1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:52:09.282738 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.282714    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-kube-api-access-zb69q" pod "373ace8c-529f-48a0-9042-c55c3bc06c1a" (UID: "373ace8c-529f-48a0-9042-c55c3bc06c1a"). InnerVolumeSpecName "kube-api-access-zb69q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:52:09.285093 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.285068    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-provider-credentials" pod "373ace8c-529f-48a0-9042-c55c3bc06c1a" (UID: "373ace8c-529f-48a0-9042-c55c3bc06c1a"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:52:09.401348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.381275    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:09.401623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.401598    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zb69q\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-kube-api-access-zb69q\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:09.401716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.401705    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373ace8c-529f-48a0-9042-c55c3bc06c1a-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:09.401806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.401796    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/373ace8c-529f-48a0-9042-c55c3bc06c1a-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:09.773855 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.773821    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"]
Nov 03 00:52:09.777614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.777588    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:09.821499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.821459    6000 generic.go:356] "Generic (PLEG): container finished" podID="373ace8c-529f-48a0-9042-c55c3bc06c1a" containerID="62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b" exitCode=0
Nov 03 00:52:09.822570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.822055    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd" event={"ID":"373ace8c-529f-48a0-9042-c55c3bc06c1a","Type":"ContainerDied","Data":"62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b"}
Nov 03 00:52:09.822807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.822784    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd" event={"ID":"373ace8c-529f-48a0-9042-c55c3bc06c1a","Type":"ContainerDied","Data":"24b8dfe2c9467b92301be44b6d467463217fd1c0bc8edbbd64f0f6ca36c85a2e"}
Nov 03 00:52:09.822988 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.822556    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"
Nov 03 00:52:09.823304 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.822817    6000 scope.go:122] "RemoveContainer" containerID="62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b"
Nov 03 00:52:09.840050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.840026    6000 scope.go:122] "RemoveContainer" containerID="62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b"
Nov 03 00:52:09.840413 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:52:09.840387    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b\": container with ID starting with 62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b not found: ID does not exist" containerID="62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b"
Nov 03 00:52:09.840527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.840423    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b"} err="failed to get container status \"62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b\": rpc error: code = NotFound desc = could not find container \"62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b\": container with ID starting with 62cd86a37b0d656f51c5ab857e85308e8a93a17658f891b397fd79b72a01849b not found: ID does not exist"
Nov 03 00:52:09.892639 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.892598    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"]
Nov 03 00:52:09.898250 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:09.898224    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd"]
Nov 03 00:52:10.157324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:10.157284    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="models-as-a-service/payload-processing-external-model-5d5f4bbcf7-jtrbd" podUID="373ace8c-529f-48a0-9042-c55c3bc06c1a" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.230:50052\" within 1s: context deadline exceeded"
Nov 03 00:52:10.157895 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:52:10.157871    6000 logging.go:55] [core] [Channel #8212 SubChannel #8213]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.230:50052", ServerName: "10.134.0.230:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.230:50052: operation was canceled"
Nov 03 00:52:10.584184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:10.584103    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373ace8c-529f-48a0-9042-c55c3bc06c1a" path="/var/lib/kubelet/pods/373ace8c-529f-48a0-9042-c55c3bc06c1a/volumes"
Nov 03 00:52:11.276984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.276952    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"]
Nov 03 00:52:11.277866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.277850    6000 cpu_manager.go:414] "containerMap: removing container" podUID="373ace8c-529f-48a0-9042-c55c3bc06c1a" containerName="payload-processing"
Nov 03 00:52:11.277939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.277868    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373ace8c-529f-48a0-9042-c55c3bc06c1a" containerName="payload-processing"
Nov 03 00:52:11.278011 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.278001    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="373ace8c-529f-48a0-9042-c55c3bc06c1a" containerName="payload-processing"
Nov 03 00:52:11.283285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.283248    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.300207 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.300184    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"]
Nov 03 00:52:11.306836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.306810    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"agc-cred-6ba878120f84\""
Nov 03 00:52:11.306836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.306829    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"routing-overlay\""
Nov 03 00:52:11.306836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.306838    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"payload-processing-external-model-e2e--977b8100-dockercfg-2tjv8\""
Nov 03 00:52:11.307057 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.306922    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"payload-processing-external-model-plugins-e2e-extmod-a-850433\""
Nov 03 00:52:11.311173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.310493    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"openshift-service-ca.crt\""
Nov 03 00:52:11.311173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.310601    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"kube-root-ca.crt\""
Nov 03 00:52:11.358479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.358435    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.359020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.358975    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.359319 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.359296    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.359772 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.359742    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjx6\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-kube-api-access-fvjx6\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.476056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.462708    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.477982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.477951    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.480967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.480941    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.481179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.481157    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjx6\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-kube-api-access-fvjx6\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.485470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.480820    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.485706 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.485239    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.490107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.490082    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.519556 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.519515    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjx6\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-kube-api-access-fvjx6\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.598851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.598758    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:11.925239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:11.925206    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"]
Nov 03 00:52:11.925913 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:52:11.925885    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcac94ec3_4fa0_4830_b58d_2e304452c921.slice/crio-661c10e14d315833b969aac251cfeb4e490731f9b5cf51ab79a22dc6fb532553 WatchSource:0}: Error finding container 661c10e14d315833b969aac251cfeb4e490731f9b5cf51ab79a22dc6fb532553: Status 404 returned error can't find the container with id 661c10e14d315833b969aac251cfeb4e490731f9b5cf51ab79a22dc6fb532553
Nov 03 00:52:12.403885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:12.403855    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"]
Nov 03 00:52:12.406783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:12.406759    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:12.869741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:12.866697    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7" event={"ID":"cac94ec3-4fa0-4830-b58d-2e304452c921","Type":"ContainerStarted","Data":"a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4"}
Nov 03 00:52:12.869741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:12.866778    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7" event={"ID":"cac94ec3-4fa0-4830-b58d-2e304452c921","Type":"ContainerStarted","Data":"661c10e14d315833b969aac251cfeb4e490731f9b5cf51ab79a22dc6fb532553"}
Nov 03 00:52:12.869741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:12.869323    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:12.909776 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:12.909723    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7" podStartSLOduration=1.909706364 podStartE2EDuration="1.909706364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:12.908530848 +0000 UTC m=+2770.463882874" watchObservedRunningTime="2026-11-03 00:52:12.909706364 +0000 UTC m=+2770.465058387"
Nov 03 00:52:14.774514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:14.774466    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq" podUID="cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" containerName="payload-processing" containerID="cri-o://3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb" gracePeriod=25
Nov 03 00:52:15.087773 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.087746    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:15.177609 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.177579    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-plugins-config-volume\") pod \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") "
Nov 03 00:52:15.177813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.177635    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-routing-overlay-volume\") pod \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") "
Nov 03 00:52:15.177813 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.177760    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-provider-credentials\") pod \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") "
Nov 03 00:52:15.178433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.177986    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-kube-api-access-xcms7\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-kube-api-access-xcms7\") pod \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\" (UID: \"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f\") "
Nov 03 00:52:15.178433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.178061    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-routing-overlay-volume" pod "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" (UID: "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:52:15.178433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.178121    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-plugins-config-volume" pod "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" (UID: "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:52:15.178752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.178683    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:15.178752 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.178709    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:15.180778 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.180752    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-kube-api-access-xcms7" pod "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" (UID: "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f"). InnerVolumeSpecName "kube-api-access-xcms7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:52:15.181061 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.181032    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-provider-credentials" pod "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" (UID: "cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:52:15.280574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.280538    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcms7\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-kube-api-access-xcms7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:15.280574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.280567    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:15.950635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.950217    6000 generic.go:356] "Generic (PLEG): container finished" podID="cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" containerID="3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb" exitCode=0
Nov 03 00:52:15.950635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.950304    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"
Nov 03 00:52:15.950635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.950337    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq" event={"ID":"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f","Type":"ContainerDied","Data":"3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb"}
Nov 03 00:52:15.950635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.950389    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq" event={"ID":"cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f","Type":"ContainerDied","Data":"537857d5eddc588f371bbcbc58adb67c078c157206a73221bdd8055edc2717f1"}
Nov 03 00:52:15.950635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.950413    6000 scope.go:122] "RemoveContainer" containerID="3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb"
Nov 03 00:52:15.964431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.964401    6000 scope.go:122] "RemoveContainer" containerID="3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb"
Nov 03 00:52:15.964805 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:52:15.964774    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb\": container with ID starting with 3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb not found: ID does not exist" containerID="3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb"
Nov 03 00:52:15.964917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:15.964819    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb"} err="failed to get container status \"3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb\": rpc error: code = NotFound desc = could not find container \"3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb\": container with ID starting with 3adff38830ed765d841e8eb3a958fc8046e1149ff233a4aa29571d1bfa879bdb not found: ID does not exist"
Nov 03 00:52:16.002083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.002050    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"]
Nov 03 00:52:16.006400 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.006374    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-955dc3/payload-processing-external-model-e2e-extmod-a-955dc3-6c96jlhfq"]
Nov 03 00:52:16.571049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.571005    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" path="/var/lib/kubelet/pods/cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f/volumes"
Nov 03 00:52:16.786087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.786053    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"]
Nov 03 00:52:16.787425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.787403    6000 cpu_manager.go:414] "containerMap: removing container" podUID="cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" containerName="payload-processing"
Nov 03 00:52:16.787490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.787429    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" containerName="payload-processing"
Nov 03 00:52:16.787743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.787729    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf90eb3f-8fcf-4ed5-b57e-f6e62754ab5f" containerName="payload-processing"
Nov 03 00:52:16.793030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.793014    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:16.812286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.812245    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-84f605-openshift-default-dockercfg-72ws6\""
Nov 03 00:52:16.830183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:16.830111    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"]
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.006915    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca3db227-6b69-4447-b265-ff64e57fc88f-istiod-ca-cert\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.007077    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-credential-socket\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.007305    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-podinfo\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.007368    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-token\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.007534    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-certs\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.007663    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-envoy\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.007692    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcqsv\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-kube-api-access-mcqsv\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.007876    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-data\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.014277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.008016    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-socket\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.109683 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.109651    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-data\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.109909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.109698    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-socket\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.109909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.109753    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca3db227-6b69-4447-b265-ff64e57fc88f-istiod-ca-cert\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.109909 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.109810    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-credential-socket\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.109954    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-podinfo\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.109997    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-token\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110115 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110080    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-certs\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110117    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-envoy\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110114    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-data\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110144    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcqsv\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-kube-api-access-mcqsv\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110155    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-socket\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110481    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-certs\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110690 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110670    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca3db227-6b69-4447-b265-ff64e57fc88f-istiod-ca-cert\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.110790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.110772    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-credential-socket\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.112659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.112622    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-envoy\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.113331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.113311    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-podinfo\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.129455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.129423    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-token\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.129660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.129496    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcqsv\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-kube-api-access-mcqsv\") pod \"e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") " pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.404216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.404107    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz" podUID="86eea21a-168f-4818-b94d-582ee02e4445" containerName="payload-processing" containerID="cri-o://35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c" gracePeriod=25
Nov 03 00:52:17.408792 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.408768    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:17.764851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.764827    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"]
Nov 03 00:52:17.765404 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:52:17.765377    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca3db227_6b69_4447_b265_ff64e57fc88f.slice/crio-4b58b76ec5fc7f565ca8dc808ba3897f7f42ad8e4913b09a8cf75bb0629a6d7d WatchSource:0}: Error finding container 4b58b76ec5fc7f565ca8dc808ba3897f7f42ad8e4913b09a8cf75bb0629a6d7d: Status 404 returned error can't find the container with id 4b58b76ec5fc7f565ca8dc808ba3897f7f42ad8e4913b09a8cf75bb0629a6d7d
Nov 03 00:52:17.769039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.769007    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:52:17.769123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.769071    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:52:17.769123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.769118    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:52:17.797886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.797859    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:17.825793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.825772    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-provider-credentials\") pod \"86eea21a-168f-4818-b94d-582ee02e4445\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") "
Nov 03 00:52:17.825955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.825922    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-plugins-config-volume\") pod \"86eea21a-168f-4818-b94d-582ee02e4445\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") "
Nov 03 00:52:17.826054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.826040    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-routing-overlay-volume\") pod \"86eea21a-168f-4818-b94d-582ee02e4445\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") "
Nov 03 00:52:17.826116 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.826083    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-kube-api-access-8xjm5\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-kube-api-access-8xjm5\") pod \"86eea21a-168f-4818-b94d-582ee02e4445\" (UID: \"86eea21a-168f-4818-b94d-582ee02e4445\") "
Nov 03 00:52:17.826414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.826384    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-plugins-config-volume" pod "86eea21a-168f-4818-b94d-582ee02e4445" (UID: "86eea21a-168f-4818-b94d-582ee02e4445"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:52:17.826574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.826554    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-routing-overlay-volume" pod "86eea21a-168f-4818-b94d-582ee02e4445" (UID: "86eea21a-168f-4818-b94d-582ee02e4445"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:52:17.826997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.826977    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:17.827083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.827007    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86eea21a-168f-4818-b94d-582ee02e4445-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:17.828439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.828419    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-provider-credentials" pod "86eea21a-168f-4818-b94d-582ee02e4445" (UID: "86eea21a-168f-4818-b94d-582ee02e4445"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:52:17.828592 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.828575    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-kube-api-access-8xjm5" pod "86eea21a-168f-4818-b94d-582ee02e4445" (UID: "86eea21a-168f-4818-b94d-582ee02e4445"). InnerVolumeSpecName "kube-api-access-8xjm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:52:17.928292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.928241    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xjm5\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-kube-api-access-8xjm5\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:17.928292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:17.928294    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86eea21a-168f-4818-b94d-582ee02e4445-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:52:18.042622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.042528    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" event={"ID":"ca3db227-6b69-4447-b265-ff64e57fc88f","Type":"ContainerStarted","Data":"b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c"}
Nov 03 00:52:18.042622 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.042586    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" event={"ID":"ca3db227-6b69-4447-b265-ff64e57fc88f","Type":"ContainerStarted","Data":"4b58b76ec5fc7f565ca8dc808ba3897f7f42ad8e4913b09a8cf75bb0629a6d7d"}
Nov 03 00:52:18.044059 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.044029    6000 generic.go:356] "Generic (PLEG): container finished" podID="86eea21a-168f-4818-b94d-582ee02e4445" containerID="35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c" exitCode=0
Nov 03 00:52:18.044196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.044068    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"
Nov 03 00:52:18.044196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.044103    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz" event={"ID":"86eea21a-168f-4818-b94d-582ee02e4445","Type":"ContainerDied","Data":"35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c"}
Nov 03 00:52:18.044196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.044156    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz" event={"ID":"86eea21a-168f-4818-b94d-582ee02e4445","Type":"ContainerDied","Data":"4f59730f930cae805ac2cf067c4f0223ac6e757be1f38e87a69a10a1e6b12704"}
Nov 03 00:52:18.044196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.044178    6000 scope.go:122] "RemoveContainer" containerID="35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c"
Nov 03 00:52:18.057091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.057070    6000 scope.go:122] "RemoveContainer" containerID="35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c"
Nov 03 00:52:18.057454 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:52:18.057434    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c\": container with ID starting with 35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c not found: ID does not exist" containerID="35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c"
Nov 03 00:52:18.057526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.057463    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c"} err="failed to get container status \"35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c\": rpc error: code = NotFound desc = could not find container \"35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c\": container with ID starting with 35ae22a6b077e3deca12a168b0eec22aaed3ad918360ec50cfd7d1dbf7288a3c not found: ID does not exist"
Nov 03 00:52:18.088788 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.088727    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podStartSLOduration=2.088713121 podStartE2EDuration="2.088713121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:18.084421761 +0000 UTC m=+2775.639773797" watchObservedRunningTime="2026-11-03 00:52:18.088713121 +0000 UTC m=+2775.644065144"
Nov 03 00:52:18.122862 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.122824    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"]
Nov 03 00:52:18.128650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.128622    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-909268/payload-processing-external-model-e2e-extmod-b-909268-6556tl8mz"]
Nov 03 00:52:18.409167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.409131    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:18.415292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.415234    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:18.569887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:18.569855    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86eea21a-168f-4818-b94d-582ee02e4445" path="/var/lib/kubelet/pods/86eea21a-168f-4818-b94d-582ee02e4445/volumes"
Nov 03 00:52:19.053438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:19.053404    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:19.054486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:19.054465    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:52:19.826395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:19.826367    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:52:19.907299 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:52:19.904017    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:52:22.263312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.263272    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"]
Nov 03 00:52:22.264179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.264162    6000 cpu_manager.go:414] "containerMap: removing container" podUID="86eea21a-168f-4818-b94d-582ee02e4445" containerName="payload-processing"
Nov 03 00:52:22.264221 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.264181    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86eea21a-168f-4818-b94d-582ee02e4445" containerName="payload-processing"
Nov 03 00:52:22.264416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.264403    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="86eea21a-168f-4818-b94d-582ee02e4445" containerName="payload-processing"
Nov 03 00:52:22.269535 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.269520    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:22.289552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.289525    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"]
Nov 03 00:52:22.295623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.295592    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-84f605\""
Nov 03 00:52:22.311859 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.311832    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-84f605-dockercfg-hwpvp\""
Nov 03 00:52:22.465286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.464877    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a89420-6064-470b-a029-d349ee66dff6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:22.465286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.465093    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv9m8\" (UniqueName: \"kubernetes.io/projected/a5a89420-6064-470b-a029-d349ee66dff6-kube-api-access-sv9m8\") pod \"payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:22.580168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.580073    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv9m8\" (UniqueName: \"kubernetes.io/projected/a5a89420-6064-470b-a029-d349ee66dff6-kube-api-access-sv9m8\") pod \"payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:22.599822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.599796    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a89420-6064-470b-a029-d349ee66dff6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:22.600925 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.600695    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a89420-6064-470b-a029-d349ee66dff6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:22.605858 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.605835    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv9m8\" (UniqueName: \"kubernetes.io/projected/a5a89420-6064-470b-a029-d349ee66dff6-kube-api-access-sv9m8\") pod \"payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:22.885178 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:22.885150    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:23.210808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:23.210778    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"]
Nov 03 00:52:23.879246 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:23.879219    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:52:24.093206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:24.093171    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n" event={"ID":"a5a89420-6064-470b-a029-d349ee66dff6","Type":"ContainerStarted","Data":"fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d"}
Nov 03 00:52:24.093416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:24.093222    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n" event={"ID":"a5a89420-6064-470b-a029-d349ee66dff6","Type":"ContainerStarted","Data":"179d3a292d11400d7a304236950dadbc23632b41a3c565a4cb0097abe5bc609b"}
Nov 03 00:52:24.093416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:24.093280    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:24.132030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:24.131929    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n" podStartSLOduration=2.13191426 podStartE2EDuration="2.13191426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:52:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:52:24.130386218 +0000 UTC m=+2781.685738245" watchObservedRunningTime="2026-11-03 00:52:24.13191426 +0000 UTC m=+2781.687266334"
Nov 03 00:52:35.121468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:52:35.121013    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 00:52:49.897463 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:52:49.897358    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:53:02.572982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:02.572953    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:53:05.232037 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.232001    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"]
Nov 03 00:53:05.237406 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.237385    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.263634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.263599    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-84f605-serving-cert\""
Nov 03 00:53:05.270668 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.270619    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"]
Nov 03 00:53:05.426545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.426508    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-metrics-certs\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.426733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.426660    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b411ea47-f364-4198-a54a-202168abadc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.426733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.426710    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv94h\" (UniqueName: \"kubernetes.io/projected/b411ea47-f364-4198-a54a-202168abadc6-kube-api-access-kv94h\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.426833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.426807    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.544746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.539648    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-metrics-certs\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.544746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.539782    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b411ea47-f364-4198-a54a-202168abadc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.544746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.539843    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv94h\" (UniqueName: \"kubernetes.io/projected/b411ea47-f364-4198-a54a-202168abadc6-kube-api-access-kv94h\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.544746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.539876    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.554242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.554208    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-maas-api-tls\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.572844 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.572783    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-metrics-certs\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.574686 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.574644    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b411ea47-f364-4198-a54a-202168abadc6-openshift-service-ca\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.580021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.579721    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv94h\" (UniqueName: \"kubernetes.io/projected/b411ea47-f364-4198-a54a-202168abadc6-kube-api-access-kv94h\") pod \"maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:05.853304 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:05.853244    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:06.160274 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:53:06.160230    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb411ea47_f364_4198_a54a_202168abadc6.slice/crio-8667af8a6b7dc43be66a7f55f465668ab773b5a87c58927e522e9caf6bf96c67 WatchSource:0}: Error finding container 8667af8a6b7dc43be66a7f55f465668ab773b5a87c58927e522e9caf6bf96c67: Status 404 returned error can't find the container with id 8667af8a6b7dc43be66a7f55f465668ab773b5a87c58927e522e9caf6bf96c67
Nov 03 00:53:06.161174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:06.161150    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"]
Nov 03 00:53:06.161849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:06.161832    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:53:06.616902 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:06.616847    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc" event={"ID":"b411ea47-f364-4198-a54a-202168abadc6","Type":"ContainerStarted","Data":"8667af8a6b7dc43be66a7f55f465668ab773b5a87c58927e522e9caf6bf96c67"}
Nov 03 00:53:07.626324 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:07.626288    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc" event={"ID":"b411ea47-f364-4198-a54a-202168abadc6","Type":"ContainerStarted","Data":"3399cadd7194e827edacb9ad3121cf2121aa53241af165d65c1f370dbb72c9a6"}
Nov 03 00:53:07.626749 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:07.626349    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:07.667725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:07.667664    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc" podStartSLOduration=2.171385374 podStartE2EDuration="2.667646555s" totalImagesPullingTime="496.261181ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:53:07.667427467 +0000 UTC m=+2825.222779491" watchObservedRunningTime="2026-11-03 00:53:07.667646555 +0000 UTC m=+2825.222998583"
Nov 03 00:53:13.641135 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:13.641106    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 00:53:19.898576 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:53:19.898537    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:53:35.095402 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.095365    6000 patch_prober.go:29] interesting pod/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 00:53:35.095993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.095436    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 00:53:35.881679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.881648    6000 generic.go:356] "Generic (PLEG): container finished" podID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerID="b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c" exitCode=0
Nov 03 00:53:35.881877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.881718    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" event={"ID":"ca3db227-6b69-4447-b265-ff64e57fc88f","Type":"ContainerDied","Data":"b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c"}
Nov 03 00:53:35.882147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.882133    6000 scope.go:122] "RemoveContainer" containerID="b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c"
Nov 03 00:53:35.883979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.883950    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:53:35.884064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.884020    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:53:35.884064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:35.884053    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:53:36.915736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:36.915700    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" event={"ID":"ca3db227-6b69-4447-b265-ff64e57fc88f","Type":"ContainerStarted","Data":"631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004"}
Nov 03 00:53:37.411492 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:37.411448    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:53:37.411733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:37.411713    6000 patch_prober.go:29] interesting pod/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 00:53:37.411812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:37.411764    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 00:53:38.411921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:38.411879    6000 patch_prober.go:29] interesting pod/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 00:53:38.412478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:38.411958    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 00:53:39.424997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:39.424957    6000 patch_prober.go:29] interesting pod/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 00:53:39.425644 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:39.425610    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 00:53:40.411689 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:40.411642    6000 patch_prober.go:29] interesting pod/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 00:53:40.411896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:40.411724    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 00:53:41.411283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:41.409645    6000 patch_prober.go:29] interesting pod/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 00:53:41.411283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:41.409721    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 00:53:42.409627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:42.409591    6000 patch_prober.go:29] interesting pod/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused" start-of-body=
Nov 03 00:53:42.409838 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:42.409650    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.243:15021/healthz/ready\": dial tcp 10.134.0.243:15021: connect: connection refused"
Nov 03 00:53:43.413872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:43.413841    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:53:43.414382 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:43.414363    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:53:43.415125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:43.415099    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 00:53:49.897978 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:53:49.897945    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:53:57.691536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:53:57.691501    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:54:04.580747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.580716    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"]
Nov 03 00:54:04.584641 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.584614    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:04.605961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.605930    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"]
Nov 03 00:54:04.611128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.611099    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-726205\""
Nov 03 00:54:04.642466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.642436    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"]
Nov 03 00:54:04.647332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.647292    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:04.662819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.662788    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"]
Nov 03 00:54:04.673198 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.673171    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-726205-dockercfg-dqd88\""
Nov 03 00:54:04.743662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.743622    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm6s4\" (UniqueName: \"kubernetes.io/projected/48bd335d-7461-4bb4-9f26-f9a4f22d9454-kube-api-access-sm6s4\") pod \"payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") " pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:04.761736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.761701    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd335d-7461-4bb4-9f26-f9a4f22d9454-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") " pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:04.769292 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.769235    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d27238b-448c-44e1-b684-fee5f43d838c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:04.815410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.815372    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f95k7\" (UniqueName: \"kubernetes.io/projected/2d27238b-448c-44e1-b684-fee5f43d838c-kube-api-access-f95k7\") pod \"payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:04.917516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.917478    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm6s4\" (UniqueName: \"kubernetes.io/projected/48bd335d-7461-4bb4-9f26-f9a4f22d9454-kube-api-access-sm6s4\") pod \"payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") " pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:04.917741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.917588    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd335d-7461-4bb4-9f26-f9a4f22d9454-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") " pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:04.917741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.917686    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d27238b-448c-44e1-b684-fee5f43d838c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:04.917883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.917745    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f95k7\" (UniqueName: \"kubernetes.io/projected/2d27238b-448c-44e1-b684-fee5f43d838c-kube-api-access-f95k7\") pod \"payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:04.919822 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.919793    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d27238b-448c-44e1-b684-fee5f43d838c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:04.939798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.939769    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm6s4\" (UniqueName: \"kubernetes.io/projected/48bd335d-7461-4bb4-9f26-f9a4f22d9454-kube-api-access-sm6s4\") pod \"payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") " pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:04.941454 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.941422    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd335d-7461-4bb4-9f26-f9a4f22d9454-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") " pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:04.942984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.942926    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f95k7\" (UniqueName: \"kubernetes.io/projected/2d27238b-448c-44e1-b684-fee5f43d838c-kube-api-access-f95k7\") pod \"payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:04.968055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:04.968013    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:05.198721 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:05.198633    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:05.275085 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:05.275062    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"]
Nov 03 00:54:05.279400 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:54:05.279366    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d27238b_448c_44e1_b684_fee5f43d838c.slice/crio-dd114b3f24a59174dfa0ea6b7c791f7326731fc516b175ffb22826e0cda54e8f WatchSource:0}: Error finding container dd114b3f24a59174dfa0ea6b7c791f7326731fc516b175ffb22826e0cda54e8f: Status 404 returned error can't find the container with id dd114b3f24a59174dfa0ea6b7c791f7326731fc516b175ffb22826e0cda54e8f
Nov 03 00:54:05.537015 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:54:05.536981    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48bd335d_7461_4bb4_9f26_f9a4f22d9454.slice/crio-4671ef5143a4953c8924211fe719874d7da54d9a432f477c82642b207a9fb3a2 WatchSource:0}: Error finding container 4671ef5143a4953c8924211fe719874d7da54d9a432f477c82642b207a9fb3a2: Status 404 returned error can't find the container with id 4671ef5143a4953c8924211fe719874d7da54d9a432f477c82642b207a9fb3a2
Nov 03 00:54:05.538328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:05.538304    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"]
Nov 03 00:54:06.242021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.241973    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" event={"ID":"2d27238b-448c-44e1-b684-fee5f43d838c","Type":"ContainerStarted","Data":"3cab8aadb71151633141c107fa71a8c5b607fff31f0e672f019eb28d91fa23cc"}
Nov 03 00:54:06.243099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.242084    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" event={"ID":"2d27238b-448c-44e1-b684-fee5f43d838c","Type":"ContainerStarted","Data":"dd114b3f24a59174dfa0ea6b7c791f7326731fc516b175ffb22826e0cda54e8f"}
Nov 03 00:54:06.243099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.242244    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:06.244197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.244157    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" event={"ID":"48bd335d-7461-4bb4-9f26-f9a4f22d9454","Type":"ContainerStarted","Data":"7556a21b1c2c9200956f0fe7059ae0ab352701b7723f3422c92d29c30d197dd4"}
Nov 03 00:54:06.244357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.244203    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" event={"ID":"48bd335d-7461-4bb4-9f26-f9a4f22d9454","Type":"ContainerStarted","Data":"4671ef5143a4953c8924211fe719874d7da54d9a432f477c82642b207a9fb3a2"}
Nov 03 00:54:06.244357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.244308    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:06.282450 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.282399    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" podStartSLOduration=2.2823856 podStartE2EDuration="2.2823856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:06.278131437 +0000 UTC m=+2883.833483453" watchObservedRunningTime="2026-11-03 00:54:06.2823856 +0000 UTC m=+2883.837737623"
Nov 03 00:54:06.315118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:06.315078    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" podStartSLOduration=2.315066206 podStartE2EDuration="2.315066206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:06.313144225 +0000 UTC m=+2883.868496251" watchObservedRunningTime="2026-11-03 00:54:06.315066206 +0000 UTC m=+2883.870418238"
Nov 03 00:54:17.270667 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:17.270523    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 00:54:17.271101 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:17.271034    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 00:54:19.898066 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:54:19.898022    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:54:25.595638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:25.595604    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"]
Nov 03 00:54:30.596263 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.596225    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6" podUID="88744866-3eaf-44ee-bc4b-4e5504901b06" containerName="payload-processing" containerID="cri-o://01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0" gracePeriod=25
Nov 03 00:54:30.928676 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.928648    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:54:30.962469 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.962441    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-plugins-config-volume\") pod \"88744866-3eaf-44ee-bc4b-4e5504901b06\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") "
Nov 03 00:54:30.962634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.962557    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-routing-overlay-volume\") pod \"88744866-3eaf-44ee-bc4b-4e5504901b06\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") "
Nov 03 00:54:30.962634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.962606    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-provider-credentials\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-provider-credentials\") pod \"88744866-3eaf-44ee-bc4b-4e5504901b06\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") "
Nov 03 00:54:30.962760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.962672    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-kube-api-access-2mlx9\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-kube-api-access-2mlx9\") pod \"88744866-3eaf-44ee-bc4b-4e5504901b06\" (UID: \"88744866-3eaf-44ee-bc4b-4e5504901b06\") "
Nov 03 00:54:30.962955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.962923    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-plugins-config-volume" pod "88744866-3eaf-44ee-bc4b-4e5504901b06" (UID: "88744866-3eaf-44ee-bc4b-4e5504901b06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:30.962955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.962919    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-routing-overlay-volume" pod "88744866-3eaf-44ee-bc4b-4e5504901b06" (UID: "88744866-3eaf-44ee-bc4b-4e5504901b06"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:30.963438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.963412    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:30.963438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.963434    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/88744866-3eaf-44ee-bc4b-4e5504901b06-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:30.965350 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.965326    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-provider-credentials" pod "88744866-3eaf-44ee-bc4b-4e5504901b06" (UID: "88744866-3eaf-44ee-bc4b-4e5504901b06"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:30.965447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:30.965358    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-kube-api-access-2mlx9" pod "88744866-3eaf-44ee-bc4b-4e5504901b06" (UID: "88744866-3eaf-44ee-bc4b-4e5504901b06"). InnerVolumeSpecName "kube-api-access-2mlx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:31.065092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.065059    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mlx9\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-kube-api-access-2mlx9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:31.065092 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.065087    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/88744866-3eaf-44ee-bc4b-4e5504901b06-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:31.472961 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.472923    6000 generic.go:356] "Generic (PLEG): container finished" podID="88744866-3eaf-44ee-bc4b-4e5504901b06" containerID="01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0" exitCode=0
Nov 03 00:54:31.473169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.472977    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"
Nov 03 00:54:31.473169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.473012    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6" event={"ID":"88744866-3eaf-44ee-bc4b-4e5504901b06","Type":"ContainerDied","Data":"01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0"}
Nov 03 00:54:31.473169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.473073    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6" event={"ID":"88744866-3eaf-44ee-bc4b-4e5504901b06","Type":"ContainerDied","Data":"530f0ef4ce791c1b20cadc0bb67f2ed75d6567cd2a02aa850fafa43240adf46d"}
Nov 03 00:54:31.473169 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.473101    6000 scope.go:122] "RemoveContainer" containerID="01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0"
Nov 03 00:54:31.484954 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.484935    6000 scope.go:122] "RemoveContainer" containerID="01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0"
Nov 03 00:54:31.485223 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:54:31.485202    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0\": container with ID starting with 01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0 not found: ID does not exist" containerID="01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0"
Nov 03 00:54:31.485294 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.485233    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0"} err="failed to get container status \"01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0\": rpc error: code = NotFound desc = could not find container \"01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0\": container with ID starting with 01c6a9fd0f2afdd51463b0e25825ee285f78635f983b13ad8141d36d904a82f0 not found: ID does not exist"
Nov 03 00:54:31.525984 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.525950    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"]
Nov 03 00:54:31.535716 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:31.535690    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-6c99z7ks6"]
Nov 03 00:54:32.623780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:32.623730    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88744866-3eaf-44ee-bc4b-4e5504901b06" path="/var/lib/kubelet/pods/88744866-3eaf-44ee-bc4b-4e5504901b06/volumes"
Nov 03 00:54:35.602904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.602863    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"]
Nov 03 00:54:35.603912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.603891    6000 cpu_manager.go:414] "containerMap: removing container" podUID="88744866-3eaf-44ee-bc4b-4e5504901b06" containerName="payload-processing"
Nov 03 00:54:35.603912 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.603913    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88744866-3eaf-44ee-bc4b-4e5504901b06" containerName="payload-processing"
Nov 03 00:54:35.604066 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.604057    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="88744866-3eaf-44ee-bc4b-4e5504901b06" containerName="payload-processing"
Nov 03 00:54:35.608111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.608090    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.625279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.625236    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"]
Nov 03 00:54:35.632618 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.632589    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"agc-cred-7886777bc4e8\""
Nov 03 00:54:35.632806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.632595    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"kube-root-ca.crt\""
Nov 03 00:54:35.632806 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.632597    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"payload-processing-external-model-plugins-e2e-extmod-a-d87781\""
Nov 03 00:54:35.632997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.632664    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"openshift-service-ca.crt\""
Nov 03 00:54:35.636398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.636355    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"routing-overlay\""
Nov 03 00:54:35.636536 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.636396    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d87781\"/\"payload-processing-external-model-e2e--2422cafc-dockercfg-7slhl\""
Nov 03 00:54:35.668597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.668563    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klpsv\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-kube-api-access-klpsv\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.668886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.668860    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.668972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.668902    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.668972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.668935    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.797505 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.797436    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:54:35.803277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.803227    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klpsv\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-kube-api-access-klpsv\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.804307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.804280    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.805403 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.805382    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.806196 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.806176    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.807175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.806117    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.807358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.805322    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.813171 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.813149    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.861332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.855718    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klpsv\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-kube-api-access-klpsv\") pod \"payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") " pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:35.922866 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:35.922834    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:36.244201 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:54:36.244172    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ee4d1f4_eb3f_4eeb_a635_e0d2f9f0a722.slice/crio-ba656185e449d935deae9cb1ae25d62e194fe75e5efb2a69a5736fd31d74e501 WatchSource:0}: Error finding container ba656185e449d935deae9cb1ae25d62e194fe75e5efb2a69a5736fd31d74e501: Status 404 returned error can't find the container with id ba656185e449d935deae9cb1ae25d62e194fe75e5efb2a69a5736fd31d74e501
Nov 03 00:54:36.244848 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.244749    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"]
Nov 03 00:54:36.603606 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.603565    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h" event={"ID":"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722","Type":"ContainerStarted","Data":"d8530c9ef636ddf4d6a49c02d0e9299b40eba610e33e5939b0e2d735aaf939f7"}
Nov 03 00:54:36.604225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.603637    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h" event={"ID":"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722","Type":"ContainerStarted","Data":"ba656185e449d935deae9cb1ae25d62e194fe75e5efb2a69a5736fd31d74e501"}
Nov 03 00:54:36.604225 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.603669    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:36.616016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.615991    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"]
Nov 03 00:54:36.620362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.620343    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.631416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.631359    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h" podStartSLOduration=1.631240983 podStartE2EDuration="1.631240983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:36.630830698 +0000 UTC m=+2914.186182725" watchObservedRunningTime="2026-11-03 00:54:36.631240983 +0000 UTC m=+2914.186593008"
Nov 03 00:54:36.636216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.636194    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"openshift-service-ca.crt\""
Nov 03 00:54:36.636541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.636521    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"payload-processing-external-model-plugins-e2e-extmod-b-d4d8c2\""
Nov 03 00:54:36.641819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.641757    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"]
Nov 03 00:54:36.642351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.642325    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"routing-overlay\""
Nov 03 00:54:36.642463 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.642332    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"kube-root-ca.crt\""
Nov 03 00:54:36.642463 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.642396    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"agc-cred-293b75b9f572\""
Nov 03 00:54:36.642670 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.642648    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-d4d8c2\"/\"payload-processing-external-model-e2e--b53073b3-dockercfg-fvcf6\""
Nov 03 00:54:36.729460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.729428    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnxt9\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-kube-api-access-vnxt9\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.729632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.729495    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.729632 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.729597    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.729717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.729691    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.835502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.835452    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnxt9\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-kube-api-access-vnxt9\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.844821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.844788    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.846050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.846025    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.855847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.855776    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.866557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.845893    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.867083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.856663    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.886285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.881890    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.909295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.888955    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnxt9\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-kube-api-access-vnxt9\") pod \"payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") " pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:36.939922 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:36.939883    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:37.279039 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:37.279009    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"]
Nov 03 00:54:37.603974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:37.603927    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g" event={"ID":"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd","Type":"ContainerStarted","Data":"131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8"}
Nov 03 00:54:37.604481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:37.603990    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g" event={"ID":"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd","Type":"ContainerStarted","Data":"16c1e870832fe86be7f9471359c94714232c92901d5f58b7016aa8f5d7704058"}
Nov 03 00:54:37.604481 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:37.604047    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:37.645300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:37.645237    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g" podStartSLOduration=1.645221567 podStartE2EDuration="1.645221567s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:37.641983644 +0000 UTC m=+2915.197335680" watchObservedRunningTime="2026-11-03 00:54:37.645221567 +0000 UTC m=+2915.200573618"
Nov 03 00:54:40.608612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:40.608583    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"]
Nov 03 00:54:41.418320 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:41.418286    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"]
Nov 03 00:54:41.421802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:41.421780    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:45.609708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:45.609671    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g" podUID="0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" containerName="payload-processing" containerID="cri-o://131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8" gracePeriod=25
Nov 03 00:54:45.898203 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:45.898176    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:46.048638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.048608    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-kube-api-access-vnxt9\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-kube-api-access-vnxt9\") pod \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") "
Nov 03 00:54:46.048823 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.048668    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-routing-overlay-volume\") pod \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") "
Nov 03 00:54:46.048867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.048852    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-plugins-config-volume\") pod \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") "
Nov 03 00:54:46.049005 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.048987    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-provider-credentials\") pod \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\" (UID: \"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd\") "
Nov 03 00:54:46.049146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.049123    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-routing-overlay-volume" pod "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" (UID: "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:46.049242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.049215    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-plugins-config-volume" pod "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" (UID: "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:46.049611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.049593    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:46.049685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.049614    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:46.051078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.051056    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-kube-api-access-vnxt9" pod "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" (UID: "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd"). InnerVolumeSpecName "kube-api-access-vnxt9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:46.051189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.051155    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-provider-credentials" pod "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" (UID: "0d49be90-3ff9-42c3-827a-bb98c4bfcbbd"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:46.151210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.151134    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:46.151210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.151163    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnxt9\" (UniqueName: \"kubernetes.io/projected/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd-kube-api-access-vnxt9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:46.419201 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.419112    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h" podUID="4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" containerName="payload-processing" containerID="cri-o://d8530c9ef636ddf4d6a49c02d0e9299b40eba610e33e5939b0e2d735aaf939f7" gracePeriod=25
Nov 03 00:54:46.688950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.688912    6000 generic.go:356] "Generic (PLEG): container finished" podID="0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" containerID="131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8" exitCode=0
Nov 03 00:54:46.689414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.688966    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"
Nov 03 00:54:46.689414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.688997    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g" event={"ID":"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd","Type":"ContainerDied","Data":"131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8"}
Nov 03 00:54:46.689414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.689058    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g" event={"ID":"0d49be90-3ff9-42c3-827a-bb98c4bfcbbd","Type":"ContainerDied","Data":"16c1e870832fe86be7f9471359c94714232c92901d5f58b7016aa8f5d7704058"}
Nov 03 00:54:46.689414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.689093    6000 scope.go:122] "RemoveContainer" containerID="131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8"
Nov 03 00:54:46.691831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.691806    6000 generic.go:356] "Generic (PLEG): container finished" podID="4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" containerID="d8530c9ef636ddf4d6a49c02d0e9299b40eba610e33e5939b0e2d735aaf939f7" exitCode=0
Nov 03 00:54:46.691973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.691878    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h" event={"ID":"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722","Type":"ContainerDied","Data":"d8530c9ef636ddf4d6a49c02d0e9299b40eba610e33e5939b0e2d735aaf939f7"}
Nov 03 00:54:46.723604 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.723589    6000 scope.go:122] "RemoveContainer" containerID="131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8"
Nov 03 00:54:46.723883 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:54:46.723860    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8\": container with ID starting with 131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8 not found: ID does not exist" containerID="131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8"
Nov 03 00:54:46.723950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.723891    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8"} err="failed to get container status \"131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8\": rpc error: code = NotFound desc = could not find container \"131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8\": container with ID starting with 131c3ab5a065c599c486315447fb2f79f4667ae5e76b1a37699b3ef55ef662e8 not found: ID does not exist"
Nov 03 00:54:46.726660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.726635    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:46.756214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.756190    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"]
Nov 03 00:54:46.767612 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.767586    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-d4d8c2/payload-processing-external-model-e2e-extmod-b-d4d8c2-54b6sn58g"]
Nov 03 00:54:46.862795 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.862767    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-provider-credentials\") pod \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") "
Nov 03 00:54:46.863021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.862826    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-plugins-config-volume\") pod \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") "
Nov 03 00:54:46.863085 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.863052    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-routing-overlay-volume\") pod \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") "
Nov 03 00:54:46.863143 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.863096    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-kube-api-access-klpsv\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-kube-api-access-klpsv\") pod \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\" (UID: \"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722\") "
Nov 03 00:54:46.863290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.863231    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-plugins-config-volume" pod "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" (UID: "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:46.863487 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.863462    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-routing-overlay-volume" pod "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" (UID: "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:54:46.863743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.863700    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:46.863743 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.863724    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:46.865448 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.865421    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-provider-credentials" pod "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" (UID: "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:46.865574 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.865455    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-kube-api-access-klpsv" pod "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" (UID: "4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722"). InnerVolumeSpecName "kube-api-access-klpsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:54:46.964929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.964848    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:46.964929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:46.964877    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klpsv\" (UniqueName: \"kubernetes.io/projected/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722-kube-api-access-klpsv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:54:47.703113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:47.703082    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"
Nov 03 00:54:47.703643 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:47.703078    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h" event={"ID":"4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722","Type":"ContainerDied","Data":"ba656185e449d935deae9cb1ae25d62e194fe75e5efb2a69a5736fd31d74e501"}
Nov 03 00:54:47.703643 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:47.703221    6000 scope.go:122] "RemoveContainer" containerID="d8530c9ef636ddf4d6a49c02d0e9299b40eba610e33e5939b0e2d735aaf939f7"
Nov 03 00:54:47.752359 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:47.752327    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"]
Nov 03 00:54:47.763534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:47.763507    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d87781/payload-processing-external-model-e2e-extmod-a-d87781-bdfbqzt6h"]
Nov 03 00:54:48.308753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.308698    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"]
Nov 03 00:54:48.310173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.310143    6000 cpu_manager.go:414] "containerMap: removing container" podUID="4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" containerName="payload-processing"
Nov 03 00:54:48.310173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.310172    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" containerName="payload-processing"
Nov 03 00:54:48.310427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.310190    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" containerName="payload-processing"
Nov 03 00:54:48.310427 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.310199    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" containerName="payload-processing"
Nov 03 00:54:48.310557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.310448    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" containerName="payload-processing"
Nov 03 00:54:48.310557 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.310462    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" containerName="payload-processing"
Nov 03 00:54:48.316030 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.316006    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.331239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.331211    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-883a9e-openshift-default-dockercfg-zml5p\""
Nov 03 00:54:48.347416 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.347388    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"]
Nov 03 00:54:48.396981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.396947    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-token\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397168 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397112    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-data\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397204    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-podinfo\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397376 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397357    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-envoy\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397404    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-certs\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397431    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istiod-ca-cert\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397494    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-socket\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397605    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs45t\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-kube-api-access-vs45t\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.397677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.397662    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-credential-socket\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.512754    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-envoy\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.512843    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-certs\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.512869    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istiod-ca-cert\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.512920    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-socket\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.513001    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs45t\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-kube-api-access-vs45t\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.513045    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-credential-socket\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.513135    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-token\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.513275    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-data\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.518420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.513345    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-podinfo\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.532289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.531785    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-certs\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.536287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.533011    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istiod-ca-cert\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.536287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.533523    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-socket\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.536287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.534191    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-credential-socket\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.536287 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.534712    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-data\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.578336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.572363    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-podinfo\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.589290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.587624    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-envoy\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.593277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.592010    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-token\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.593277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.592954    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs45t\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-kube-api-access-vs45t\") pod \"e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") " pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.662224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.646634    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:48.679054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.678469    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d49be90-3ff9-42c3-827a-bb98c4bfcbbd" path="/var/lib/kubelet/pods/0d49be90-3ff9-42c3-827a-bb98c4bfcbbd/volumes"
Nov 03 00:54:48.679307 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.679274    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722" path="/var/lib/kubelet/pods/4ee4d1f4-eb3f-4eeb-a635-e0d2f9f0a722/volumes"
Nov 03 00:54:48.748237 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.744493    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"]
Nov 03 00:54:48.770306 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.768657    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.805279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.795866    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-726205-serving-cert\""
Nov 03 00:54:48.819388 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.819350    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"]
Nov 03 00:54:48.848512 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.848470    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.848688 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.848620    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stkc9\" (UniqueName: \"kubernetes.io/projected/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-kube-api-access-stkc9\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.848916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.848717    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.848916 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.848904    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-metrics-certs\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.950313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.950275    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-metrics-certs\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.950544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.950523    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.950631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.950602    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stkc9\" (UniqueName: \"kubernetes.io/projected/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-kube-api-access-stkc9\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.950755 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.950708    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.950856 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:54:48.950835    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-serving-cert: secret "maas-api-e2e-inf-b-726205-serving-cert" not found
Nov 03 00:54:48.950932 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:54:48.950918    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls podName:2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d nodeName:}" failed. No retries permitted until 2026-11-03 00:54:49.4508978 +0000 UTC m=+2927.006249819 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls") pod "maas-api-e2e-inf-b-726205-6d67b7849f-scbgd" (UID: "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d") : secret "maas-api-e2e-inf-b-726205-serving-cert" not found
Nov 03 00:54:48.951155 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.951134    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-openshift-service-ca\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:48.953128 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:48.953055    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-metrics-certs\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:49.014786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.014663    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stkc9\" (UniqueName: \"kubernetes.io/projected/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-kube-api-access-stkc9\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:49.100761 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:54:49.100731    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fda859_71bf_4b39_bb5c_5a26bd7ef43a.slice/crio-20c73db07e103ec352fc29835afe9a7623faa72bcdd55b2319b519246dec4583 WatchSource:0}: Error finding container 20c73db07e103ec352fc29835afe9a7623faa72bcdd55b2319b519246dec4583: Status 404 returned error can't find the container with id 20c73db07e103ec352fc29835afe9a7623faa72bcdd55b2319b519246dec4583
Nov 03 00:54:49.101424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.101401    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"]
Nov 03 00:54:49.104035 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.104002    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:54:49.104113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.104076    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:54:49.104113 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.104107    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:54:49.458322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.458239    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:49.460872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.460843    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls\") pod \"maas-api-e2e-inf-b-726205-6d67b7849f-scbgd\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:49.734016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.733919    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:49.747880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.747838    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" event={"ID":"98fda859-71bf-4b39-bb5c-5a26bd7ef43a","Type":"ContainerStarted","Data":"3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7"}
Nov 03 00:54:49.748072 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.747938    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" event={"ID":"98fda859-71bf-4b39-bb5c-5a26bd7ef43a","Type":"ContainerStarted","Data":"20c73db07e103ec352fc29835afe9a7623faa72bcdd55b2319b519246dec4583"}
Nov 03 00:54:49.793289 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:49.793049    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podStartSLOduration=1.793034647 podStartE2EDuration="1.793034647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:49.788481342 +0000 UTC m=+2927.343833366" watchObservedRunningTime="2026-11-03 00:54:49.793034647 +0000 UTC m=+2927.348386671"
Nov 03 00:54:49.906802 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:54:49.906758    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:54:50.098333 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.097367    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"]
Nov 03 00:54:50.652799 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.652772    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:50.660056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.660032    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:50.757864 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.757828    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd" event={"ID":"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d","Type":"ContainerStarted","Data":"0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0"}
Nov 03 00:54:50.758054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.757895    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd" event={"ID":"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d","Type":"ContainerStarted","Data":"fc4bc65e3153c751b64407e30a0f4cc246b6f513cb7de677ec75436839544525"}
Nov 03 00:54:50.758054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.757928    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:54:50.758176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.758159    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:50.759170 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.759150    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:54:50.798024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:50.797931    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd" podStartSLOduration=2.421754401 podStartE2EDuration="2.797913653s" totalImagesPullingTime="376.159252ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:54:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:54:50.79671041 +0000 UTC m=+2928.352062434" watchObservedRunningTime="2026-11-03 00:54:50.797913653 +0000 UTC m=+2928.353265677"
Nov 03 00:54:56.771537 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:54:56.771509    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 00:55:06.152645 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.152604    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"]
Nov 03 00:55:06.159526 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.159500    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.175153 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.175122    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"]
Nov 03 00:55:06.181761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.181732    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-d9e067\"/\"agc-cred-3fbcdbf51fce\""
Nov 03 00:55:06.333746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.333623    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.333746 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.333685    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d974l\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-kube-api-access-d974l\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.334021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.333956    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.334088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.334034    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.435565 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.435529    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.435777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.435573    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d974l\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-kube-api-access-d974l\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.435777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.435705    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.435777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.435747    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.436468 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.436437    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.436735 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.436496    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.438709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.438686    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.454313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.454238    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d974l\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-kube-api-access-d974l\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.476851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.476824    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:06.785643 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:55:06.785608    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod922e6643_1a84_4c9d_bf5c_0c10c3a440ba.slice/crio-8d77cc31ff7e75750968046284620bdc411fa08e4ee3f96425cd4232693730be WatchSource:0}: Error finding container 8d77cc31ff7e75750968046284620bdc411fa08e4ee3f96425cd4232693730be: Status 404 returned error can't find the container with id 8d77cc31ff7e75750968046284620bdc411fa08e4ee3f96425cd4232693730be
Nov 03 00:55:06.786892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.786859    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"]
Nov 03 00:55:06.880718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.880679    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"]
Nov 03 00:55:06.886224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.886197    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:06.899722 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.899689    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-fdacdb\"/\"agc-cred-17a055a18398\""
Nov 03 00:55:06.902638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.902609    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"]
Nov 03 00:55:06.946543 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.946505    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjzrv\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-kube-api-access-vjzrv\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:06.946769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.946742    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:06.946976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.946914    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:06.946976 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.946965    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:06.972412 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.972378    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2" event={"ID":"922e6643-1a84-4c9d-bf5c-0c10c3a440ba","Type":"ContainerStarted","Data":"cce1a11dfb46631d9afdf8c3335d3f7c95e7608ff401bfae2803b52dc0b16648"}
Nov 03 00:55:06.972631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.972436    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2" event={"ID":"922e6643-1a84-4c9d-bf5c-0c10c3a440ba","Type":"ContainerStarted","Data":"8d77cc31ff7e75750968046284620bdc411fa08e4ee3f96425cd4232693730be"}
Nov 03 00:55:06.972631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:06.972486    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:07.013836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.013781    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2" podStartSLOduration=1.013766023 podStartE2EDuration="1.013766023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:07.010893462 +0000 UTC m=+2944.566245468" watchObservedRunningTime="2026-11-03 00:55:07.013766023 +0000 UTC m=+2944.569118048"
Nov 03 00:55:07.048893 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.048811    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.049054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.048932    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.049054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.048983    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.049150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.049121    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjzrv\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-kube-api-access-vjzrv\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.049761 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.049731    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.049908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.049803    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.051955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.051932    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.067827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.067800    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjzrv\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-kube-api-access-vjzrv\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.203479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.203438    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:07.590863 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.589105    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"]
Nov 03 00:55:07.981753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.981711    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9" event={"ID":"629dfba3-dec0-43fe-93e4-665f81e14050","Type":"ContainerStarted","Data":"af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e"}
Nov 03 00:55:07.982021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.981776    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9" event={"ID":"629dfba3-dec0-43fe-93e4-665f81e14050","Type":"ContainerStarted","Data":"539e4a6eb6321f18df9e59e2d13b1d90e6db36d56cf9c14b705e66b792e0a90a"}
Nov 03 00:55:07.982021 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:07.981991    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:12.139662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:12.139594    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9" podStartSLOduration=6.139573882 podStartE2EDuration="6.139573882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:08.034689091 +0000 UTC m=+2945.590041111" watchObservedRunningTime="2026-11-03 00:55:12.139573882 +0000 UTC m=+2949.694925907"
Nov 03 00:55:12.143713 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:12.143687    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"]
Nov 03 00:55:14.073314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.073236    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"]
Nov 03 00:55:14.082354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.082311    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.105276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.105231    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"payload-processing-external-model-e2e--bf641009-dockercfg-76t7d\""
Nov 03 00:55:14.105538 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.105513    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-c738e0\"/\"agc-cred-809dd9a79e29\""
Nov 03 00:55:14.109551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.109525    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"]
Nov 03 00:55:14.168648 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.168614    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.168826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.168738    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.168903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.168881    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5wg6\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-kube-api-access-z5wg6\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.169036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.169024    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.281460 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.281422    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.281853 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.281823    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.284595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.284557    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.284917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.284898    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5wg6\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-kube-api-access-z5wg6\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.285551 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.282803    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.286326 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:14.282944    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c738e0/agc-cred-809dd9a79e29: secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:14.286429 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:14.286336    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm: secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:14.286495 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:14.286442    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials podName:169b458f-ff7f-48b9-8e51-5859a60cb49e nodeName:}" failed. No retries permitted until 2026-11-03 00:55:14.786420824 +0000 UTC m=+2952.341772843 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm" (UID: "169b458f-ff7f-48b9-8e51-5859a60cb49e") : secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:14.286789 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.286765    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.325352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.317977    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5wg6\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-kube-api-access-z5wg6\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.413283 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.410244    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"]
Nov 03 00:55:14.413283 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:14.410692    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[provider-credentials], unattached volumes=[], failed to process volumes=[]: context canceled" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm" podUID="169b458f-ff7f-48b9-8e51-5859a60cb49e"
Nov 03 00:55:14.808054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:14.808013    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:14.808874 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:14.808850    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c738e0/agc-cred-809dd9a79e29: secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:14.809005 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:14.808993    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm: secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:14.809162 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:14.809152    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials podName:169b458f-ff7f-48b9-8e51-5859a60cb49e nodeName:}" failed. No retries permitted until 2026-11-03 00:55:15.809130296 +0000 UTC m=+2953.364482317 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm" (UID: "169b458f-ff7f-48b9-8e51-5859a60cb49e") : secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:15.115281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.112776    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:15.139731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.139703    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:15.216523 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.216493    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-routing-overlay-volume\") pod \"169b458f-ff7f-48b9-8e51-5859a60cb49e\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") "
Nov 03 00:55:15.216727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.216645    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-plugins-config-volume\") pod \"169b458f-ff7f-48b9-8e51-5859a60cb49e\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") "
Nov 03 00:55:15.216808 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.216740    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-kube-api-access-z5wg6\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-kube-api-access-z5wg6\") pod \"169b458f-ff7f-48b9-8e51-5859a60cb49e\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") "
Nov 03 00:55:15.216958 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.216934    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-routing-overlay-volume" pod "169b458f-ff7f-48b9-8e51-5859a60cb49e" (UID: "169b458f-ff7f-48b9-8e51-5859a60cb49e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:15.217082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.217057    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-plugins-config-volume" pod "169b458f-ff7f-48b9-8e51-5859a60cb49e" (UID: "169b458f-ff7f-48b9-8e51-5859a60cb49e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:15.217328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.217314    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:15.217377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.217332    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/169b458f-ff7f-48b9-8e51-5859a60cb49e-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:15.219042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.218997    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-kube-api-access-z5wg6" pod "169b458f-ff7f-48b9-8e51-5859a60cb49e" (UID: "169b458f-ff7f-48b9-8e51-5859a60cb49e"). InnerVolumeSpecName "kube-api-access-z5wg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:15.319235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.319201    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5wg6\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-kube-api-access-z5wg6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:15.827736 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:15.827700    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm\" (UID: \"169b458f-ff7f-48b9-8e51-5859a60cb49e\") " pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:15.827945 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:15.827879    6000 projected.go:266] Couldn't get secret ai-tenant-e2e-extmod-a-c738e0/agc-cred-809dd9a79e29: secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:15.827945 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:15.827905    6000 projected.go:196] Error preparing data for projected volume provider-credentials for pod ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm: secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:15.828050 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:15.827964    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials podName:169b458f-ff7f-48b9-8e51-5859a60cb49e nodeName:}" failed. No retries permitted until 2026-11-03 00:55:17.827949902 +0000 UTC m=+2955.383301918 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials") pod "payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm" (UID: "169b458f-ff7f-48b9-8e51-5859a60cb49e") : secret "agc-cred-809dd9a79e29" not found
Nov 03 00:55:16.166281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.165605    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"
Nov 03 00:55:16.292098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.292060    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"]
Nov 03 00:55:16.302620 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.300087    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-5cff6bqnm"]
Nov 03 00:55:16.351028 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.350991    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/169b458f-ff7f-48b9-8e51-5859a60cb49e-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:16.479342 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.475559    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"]
Nov 03 00:55:16.492279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.482302    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.492279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.488071    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"]
Nov 03 00:55:16.497593 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.497559    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-a-850433\"/\"agc-cred-2e29c2754e72\""
Nov 03 00:55:16.763934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.710163    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.763934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.710472    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.763934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.710505    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxfr9\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-kube-api-access-jxfr9\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.763934 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.710541    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.772651 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.772127    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="169b458f-ff7f-48b9-8e51-5859a60cb49e" path="/var/lib/kubelet/pods/169b458f-ff7f-48b9-8e51-5859a60cb49e/volumes"
Nov 03 00:55:16.812647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.812609    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.812647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.812658    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxfr9\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-kube-api-access-jxfr9\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.812920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.812698    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.812920 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.812906    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.813453 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.813424    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.813698 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.813672    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.815951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.815929    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:16.831126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:16.831101    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxfr9\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-kube-api-access-jxfr9\") pod \"payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:17.086335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.086231    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:17.144437 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.144397    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg" podUID="ca666b1d-bb8c-403a-ad7d-009f8c066ead" containerName="payload-processing" containerID="cri-o://a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f" gracePeriod=25
Nov 03 00:55:17.327573 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.327543    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"]
Nov 03 00:55:17.338534 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.338452    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.354033 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.354003    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"]
Nov 03 00:55:17.356953 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.356926    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-extmod-b-ac338d\"/\"agc-cred-d46a40f20fab\""
Nov 03 00:55:17.401894 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.401865    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"]
Nov 03 00:55:17.423362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.423334    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.423600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.423577    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.423677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.423632    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.423740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.423724    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brrcd\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-kube-api-access-brrcd\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.487759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.487726    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:55:17.524568 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.524536    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-plugins-config-volume\") pod \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") "
Nov 03 00:55:17.524747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.524653    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-kube-api-access-sxd7b\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-kube-api-access-sxd7b\") pod \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") "
Nov 03 00:55:17.524947 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.524925    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-provider-credentials\") pod \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") "
Nov 03 00:55:17.525078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.525010    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-routing-overlay-volume\") pod \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\" (UID: \"ca666b1d-bb8c-403a-ad7d-009f8c066ead\") "
Nov 03 00:55:17.525305 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.525237    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-plugins-config-volume" pod "ca666b1d-bb8c-403a-ad7d-009f8c066ead" (UID: "ca666b1d-bb8c-403a-ad7d-009f8c066ead"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:17.525449 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.525349    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.525969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.525944    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.526147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.526122    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.526335 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.526206    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brrcd\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-kube-api-access-brrcd\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.526709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.526388    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.526974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.526944    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-routing-overlay-volume" pod "ca666b1d-bb8c-403a-ad7d-009f8c066ead" (UID: "ca666b1d-bb8c-403a-ad7d-009f8c066ead"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:17.527552 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.527528    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.528277 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.528238    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.529188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.529026    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.530975 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.530940    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-provider-credentials" pod "ca666b1d-bb8c-403a-ad7d-009f8c066ead" (UID: "ca666b1d-bb8c-403a-ad7d-009f8c066ead"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:17.531341 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.531315    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-kube-api-access-sxd7b" pod "ca666b1d-bb8c-403a-ad7d-009f8c066ead" (UID: "ca666b1d-bb8c-403a-ad7d-009f8c066ead"). InnerVolumeSpecName "kube-api-access-sxd7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:17.550167 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.550138    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brrcd\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-kube-api-access-brrcd\") pod \"payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") " pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.628374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.628339    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxd7b\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-kube-api-access-sxd7b\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.628374 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.628376    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/ca666b1d-bb8c-403a-ad7d-009f8c066ead-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.628627 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.628392    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/ca666b1d-bb8c-403a-ad7d-009f8c066ead-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:17.667266 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.667235    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:17.976433 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:55:17.976386    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaad8926_77d9_4ea6_bd34_00afc826fd84.slice/crio-2d2fe80e82ddecaf9849177c4d5bdec303e0f6d719b3f93dbebc9b263a1d35eb WatchSource:0}: Error finding container 2d2fe80e82ddecaf9849177c4d5bdec303e0f6d719b3f93dbebc9b263a1d35eb: Status 404 returned error can't find the container with id 2d2fe80e82ddecaf9849177c4d5bdec303e0f6d719b3f93dbebc9b263a1d35eb
Nov 03 00:55:17.976682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.976664    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"]
Nov 03 00:55:17.985439 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:17.985418    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:55:18.053087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.053058    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"]
Nov 03 00:55:18.192614 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.192578    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q" event={"ID":"aaad8926-77d9-4ea6-bd34-00afc826fd84","Type":"ContainerStarted","Data":"dcbba6b9cd8f310415e95e9117fa271224059af76659231d6040d818bf04441f"}
Nov 03 00:55:18.192814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.192642    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q" event={"ID":"aaad8926-77d9-4ea6-bd34-00afc826fd84","Type":"ContainerStarted","Data":"2d2fe80e82ddecaf9849177c4d5bdec303e0f6d719b3f93dbebc9b263a1d35eb"}
Nov 03 00:55:18.192814 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.192670    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:18.194516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.194485    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x" event={"ID":"aaaf801b-3355-49fd-84b4-742538542504","Type":"ContainerStarted","Data":"3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b"}
Nov 03 00:55:18.194696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.194561    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x" event={"ID":"aaaf801b-3355-49fd-84b4-742538542504","Type":"ContainerStarted","Data":"1bd0806f5b24132f7348889ae73a4b0e168c31e324e94897696951f363a3be3e"}
Nov 03 00:55:18.194696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.194623    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:18.197142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.197114    6000 generic.go:356] "Generic (PLEG): container finished" podID="ca666b1d-bb8c-403a-ad7d-009f8c066ead" containerID="a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f" exitCode=0
Nov 03 00:55:18.197338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.197208    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"
Nov 03 00:55:18.197338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.197221    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg" event={"ID":"ca666b1d-bb8c-403a-ad7d-009f8c066ead","Type":"ContainerDied","Data":"a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f"}
Nov 03 00:55:18.197338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.197293    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg" event={"ID":"ca666b1d-bb8c-403a-ad7d-009f8c066ead","Type":"ContainerDied","Data":"2453ddba709b2a24c469e1dff400d9498debb95e2535d4042e98963372ce2ebd"}
Nov 03 00:55:18.197338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.197321    6000 scope.go:122] "RemoveContainer" containerID="a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f"
Nov 03 00:55:18.213801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.211808    6000 scope.go:122] "RemoveContainer" containerID="a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f"
Nov 03 00:55:18.214126 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:18.214106    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f\": container with ID starting with a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f not found: ID does not exist" containerID="a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f"
Nov 03 00:55:18.214176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.214135    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f"} err="failed to get container status \"a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f\": rpc error: code = NotFound desc = could not find container \"a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f\": container with ID starting with a267bec701156089b8e88960f1ceb70294b7ab2e51aacac7f4d878172003236f not found: ID does not exist"
Nov 03 00:55:18.232216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.232139    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q" podStartSLOduration=1.2321274070000001 podStartE2EDuration="1.232127407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:18.229599987 +0000 UTC m=+2955.784952012" watchObservedRunningTime="2026-11-03 00:55:18.232127407 +0000 UTC m=+2955.787479430"
Nov 03 00:55:18.267720 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.267672    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x" podStartSLOduration=2.267658854 podStartE2EDuration="2.267658854s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:18.266021693 +0000 UTC m=+2955.821373716" watchObservedRunningTime="2026-11-03 00:55:18.267658854 +0000 UTC m=+2955.823010951"
Nov 03 00:55:18.303071 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.303042    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"]
Nov 03 00:55:18.306879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.306857    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-c738e0/payload-processing-external-model-e2e-extmod-a-c738e0-74b4q6kqg"]
Nov 03 00:55:18.570204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.570123    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca666b1d-bb8c-403a-ad7d-009f8c066ead" path="/var/lib/kubelet/pods/ca666b1d-bb8c-403a-ad7d-009f8c066ead/volumes"
Nov 03 00:55:18.994702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:18.994668    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:55:19.067146 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:19.067102    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"]
Nov 03 00:55:19.897609 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:19.897576    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:55:23.054269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.054205    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm" podUID="977794cf-6dd1-43c8-a000-0f885470cae7" containerName="payload-processing" containerID="cri-o://9a153d085b0e7b1454274e1c3c512dd1f1ceb92c9a13f7a0fe1c1282ca4ee0d7" gracePeriod=25
Nov 03 00:55:23.152183 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:55:23.152146    6000 logging.go:55] [core] [Channel #10107 SubChannel #10108]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.237:50052", ServerName: "10.134.0.237:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.237:50052: connect: connection refused"
Nov 03 00:55:23.273629 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.273593    6000 generic.go:356] "Generic (PLEG): container finished" podID="977794cf-6dd1-43c8-a000-0f885470cae7" containerID="9a153d085b0e7b1454274e1c3c512dd1f1ceb92c9a13f7a0fe1c1282ca4ee0d7" exitCode=0
Nov 03 00:55:23.273815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.273665    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm" event={"ID":"977794cf-6dd1-43c8-a000-0f885470cae7","Type":"ContainerDied","Data":"9a153d085b0e7b1454274e1c3c512dd1f1ceb92c9a13f7a0fe1c1282ca4ee0d7"}
Nov 03 00:55:23.356045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.356018    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:55:23.445717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.445674    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-plugins-config-volume\") pod \"977794cf-6dd1-43c8-a000-0f885470cae7\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") "
Nov 03 00:55:23.445905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.445767    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-kube-api-access-p4fpj\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-kube-api-access-p4fpj\") pod \"977794cf-6dd1-43c8-a000-0f885470cae7\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") "
Nov 03 00:55:23.445905 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.445837    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-routing-overlay-volume\") pod \"977794cf-6dd1-43c8-a000-0f885470cae7\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") "
Nov 03 00:55:23.446082 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.445947    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-provider-credentials\") pod \"977794cf-6dd1-43c8-a000-0f885470cae7\" (UID: \"977794cf-6dd1-43c8-a000-0f885470cae7\") "
Nov 03 00:55:23.446181 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.446131    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-plugins-config-volume" pod "977794cf-6dd1-43c8-a000-0f885470cae7" (UID: "977794cf-6dd1-43c8-a000-0f885470cae7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:23.446673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.446644    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-routing-overlay-volume" pod "977794cf-6dd1-43c8-a000-0f885470cae7" (UID: "977794cf-6dd1-43c8-a000-0f885470cae7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:23.446817 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.446793    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:23.448852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.448824    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-provider-credentials" pod "977794cf-6dd1-43c8-a000-0f885470cae7" (UID: "977794cf-6dd1-43c8-a000-0f885470cae7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:23.449110 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.449079    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-kube-api-access-p4fpj" pod "977794cf-6dd1-43c8-a000-0f885470cae7" (UID: "977794cf-6dd1-43c8-a000-0f885470cae7"). InnerVolumeSpecName "kube-api-access-p4fpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:23.548399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.548366    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4fpj\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-kube-api-access-p4fpj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:23.548399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.548396    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/977794cf-6dd1-43c8-a000-0f885470cae7-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:23.548399 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:23.548405    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/977794cf-6dd1-43c8-a000-0f885470cae7-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:24.067504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.067453    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2" podUID="c7dcb737-431f-433b-a63b-1adaac0f081e" containerName="payload-processing" containerID="cri-o://be8d286be325522b119c6abab410ce6b675745c05834bc8313a5bda4cf28b60f" gracePeriod=25
Nov 03 00:55:24.153163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.153112    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm" podUID="977794cf-6dd1-43c8-a000-0f885470cae7" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.237:50052\" within 1s: context deadline exceeded"
Nov 03 00:55:24.164403 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:55:24.164332    6000 logging.go:55] [core] [Channel #10112 SubChannel #10113]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.238:50052", ServerName: "10.134.0.238:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.238:50052: connect: connection refused"
Nov 03 00:55:24.285887 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.285852    6000 generic.go:356] "Generic (PLEG): container finished" podID="c7dcb737-431f-433b-a63b-1adaac0f081e" containerID="be8d286be325522b119c6abab410ce6b675745c05834bc8313a5bda4cf28b60f" exitCode=0
Nov 03 00:55:24.286076 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.285985    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2" event={"ID":"c7dcb737-431f-433b-a63b-1adaac0f081e","Type":"ContainerDied","Data":"be8d286be325522b119c6abab410ce6b675745c05834bc8313a5bda4cf28b60f"}
Nov 03 00:55:24.287998 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.287965    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm" event={"ID":"977794cf-6dd1-43c8-a000-0f885470cae7","Type":"ContainerDied","Data":"789cefdd87af798f267a11e52918bb8251b11a48c2e85d8998b82c5bd4242fe2"}
Nov 03 00:55:24.288145 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.288000    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"
Nov 03 00:55:24.288145 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.288019    6000 scope.go:122] "RemoveContainer" containerID="9a153d085b0e7b1454274e1c3c512dd1f1ceb92c9a13f7a0fe1c1282ca4ee0d7"
Nov 03 00:55:24.340147 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.340115    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"]
Nov 03 00:55:24.346163 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.346119    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dr6hbm"]
Nov 03 00:55:24.368704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.368676    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:55:24.469760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.469718    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-routing-overlay-volume\") pod \"c7dcb737-431f-433b-a63b-1adaac0f081e\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") "
Nov 03 00:55:24.469760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.469772    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-kube-api-access-p527w\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-kube-api-access-p527w\") pod \"c7dcb737-431f-433b-a63b-1adaac0f081e\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") "
Nov 03 00:55:24.470043 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.469898    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-provider-credentials\") pod \"c7dcb737-431f-433b-a63b-1adaac0f081e\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") "
Nov 03 00:55:24.470043 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.469978    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-plugins-config-volume\") pod \"c7dcb737-431f-433b-a63b-1adaac0f081e\" (UID: \"c7dcb737-431f-433b-a63b-1adaac0f081e\") "
Nov 03 00:55:24.470164 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.470112    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-routing-overlay-volume" pod "c7dcb737-431f-433b-a63b-1adaac0f081e" (UID: "c7dcb737-431f-433b-a63b-1adaac0f081e"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:24.470422 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.470401    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-plugins-config-volume" pod "c7dcb737-431f-433b-a63b-1adaac0f081e" (UID: "c7dcb737-431f-433b-a63b-1adaac0f081e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:24.470718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.470697    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:24.470718 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.470720    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c7dcb737-431f-433b-a63b-1adaac0f081e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:24.472190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.472169    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-kube-api-access-p527w" pod "c7dcb737-431f-433b-a63b-1adaac0f081e" (UID: "c7dcb737-431f-433b-a63b-1adaac0f081e"). InnerVolumeSpecName "kube-api-access-p527w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:24.472363 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.472276    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-provider-credentials" pod "c7dcb737-431f-433b-a63b-1adaac0f081e" (UID: "c7dcb737-431f-433b-a63b-1adaac0f081e"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:24.574833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.574795    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p527w\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-kube-api-access-p527w\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:24.575031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.574841    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/c7dcb737-431f-433b-a63b-1adaac0f081e-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:24.577993 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:24.577912    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977794cf-6dd1-43c8-a000-0f885470cae7" path="/var/lib/kubelet/pods/977794cf-6dd1-43c8-a000-0f885470cae7/volumes"
Nov 03 00:55:25.164452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:25.164402    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2" podUID="c7dcb737-431f-433b-a63b-1adaac0f081e" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.238:50052\" within 1s: context deadline exceeded"
Nov 03 00:55:25.296812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:25.296778    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"
Nov 03 00:55:25.296812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:25.296792    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2" event={"ID":"c7dcb737-431f-433b-a63b-1adaac0f081e","Type":"ContainerDied","Data":"16a6c821975ada328cd506dc3321e71176f5a164002e1ce4df658672764a0e4f"}
Nov 03 00:55:25.297078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:25.296855    6000 scope.go:122] "RemoveContainer" containerID="be8d286be325522b119c6abab410ce6b675745c05834bc8313a5bda4cf28b60f"
Nov 03 00:55:25.347162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:25.347127    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"]
Nov 03 00:55:25.358589 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:25.358566    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694fmd4f2"]
Nov 03 00:55:26.569585 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:26.569556    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7dcb737-431f-433b-a63b-1adaac0f081e" path="/var/lib/kubelet/pods/c7dcb737-431f-433b-a63b-1adaac0f081e/volumes"
Nov 03 00:55:29.247420 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:29.247368    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:55:29.250323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:29.250244    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:55:29.317008 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:29.316973    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"]
Nov 03 00:55:29.359126 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:29.359088    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"]
Nov 03 00:55:33.401367 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.401334    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"]
Nov 03 00:55:33.402244 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402228    6000 cpu_manager.go:414] "containerMap: removing container" podUID="977794cf-6dd1-43c8-a000-0f885470cae7" containerName="payload-processing"
Nov 03 00:55:33.402314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402246    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977794cf-6dd1-43c8-a000-0f885470cae7" containerName="payload-processing"
Nov 03 00:55:33.402314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402274    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c7dcb737-431f-433b-a63b-1adaac0f081e" containerName="payload-processing"
Nov 03 00:55:33.402314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402280    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7dcb737-431f-433b-a63b-1adaac0f081e" containerName="payload-processing"
Nov 03 00:55:33.402314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402314    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ca666b1d-bb8c-403a-ad7d-009f8c066ead" containerName="payload-processing"
Nov 03 00:55:33.402445 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402319    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca666b1d-bb8c-403a-ad7d-009f8c066ead" containerName="payload-processing"
Nov 03 00:55:33.402500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402473    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="977794cf-6dd1-43c8-a000-0f885470cae7" containerName="payload-processing"
Nov 03 00:55:33.402500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402481    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7dcb737-431f-433b-a63b-1adaac0f081e" containerName="payload-processing"
Nov 03 00:55:33.402500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.402494    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca666b1d-bb8c-403a-ad7d-009f8c066ead" containerName="payload-processing"
Nov 03 00:55:33.406544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.406526    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:33.432414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.432383    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-49d7e5\""
Nov 03 00:55:33.435431 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.435409    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-49d7e5-dockercfg-86hvm\""
Nov 03 00:55:33.442064 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.442037    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"]
Nov 03 00:55:33.572295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.545957    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpfj\" (UniqueName: \"kubernetes.io/projected/e049c769-a863-4f07-8100-3d5d62c681d3-kube-api-access-vjpfj\") pod \"payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:33.572295 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.546234    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049c769-a863-4f07-8100-3d5d62c681d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:33.656330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.652329    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049c769-a863-4f07-8100-3d5d62c681d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:33.656330 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.652460    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpfj\" (UniqueName: \"kubernetes.io/projected/e049c769-a863-4f07-8100-3d5d62c681d3-kube-api-access-vjpfj\") pod \"payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:33.675280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.661171    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049c769-a863-4f07-8100-3d5d62c681d3-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:33.737398 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.737365    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"]
Nov 03 00:55:33.746899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.746865    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:33.750316 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.749246    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"]
Nov 03 00:55:33.750679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.750642    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpfj\" (UniqueName: \"kubernetes.io/projected/e049c769-a863-4f07-8100-3d5d62c681d3-kube-api-access-vjpfj\") pod \"payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:33.856206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.856165    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrxg\" (UniqueName: \"kubernetes.io/projected/3fc532fe-49e8-404d-ab30-fd29bae9f031-kube-api-access-5lrxg\") pod \"payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:33.856393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.856320    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc532fe-49e8-404d-ab30-fd29bae9f031-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:33.957935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.957837    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrxg\" (UniqueName: \"kubernetes.io/projected/3fc532fe-49e8-404d-ab30-fd29bae9f031-kube-api-access-5lrxg\") pod \"payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:33.958118 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.957944    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc532fe-49e8-404d-ab30-fd29bae9f031-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:33.958706 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.958687    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc532fe-49e8-404d-ab30-fd29bae9f031-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:33.976678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:33.976652    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrxg\" (UniqueName: \"kubernetes.io/projected/3fc532fe-49e8-404d-ab30-fd29bae9f031-kube-api-access-5lrxg\") pod \"payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") " pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:34.021647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.021617    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:34.086123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.085895    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:34.318174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.318095    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf" podUID="64dd5195-46b0-4e2d-9960-7d6a6026aa04" containerName="payload-processing" containerID="cri-o://8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889" gracePeriod=25
Nov 03 00:55:34.328370 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:55:34.328340    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode049c769_a863_4f07_8100_3d5d62c681d3.slice/crio-9c430a27276e94e96c56a730efc69c7e73e87d4c4e59df6b8048dfab9741cb42 WatchSource:0}: Error finding container 9c430a27276e94e96c56a730efc69c7e73e87d4c4e59df6b8048dfab9741cb42: Status 404 returned error can't find the container with id 9c430a27276e94e96c56a730efc69c7e73e87d4c4e59df6b8048dfab9741cb42
Nov 03 00:55:34.329857 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.329833    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"]
Nov 03 00:55:34.359856 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.359826    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7" podUID="cac94ec3-4fa0-4830-b58d-2e304452c921" containerName="payload-processing" containerID="cri-o://a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4" gracePeriod=25
Nov 03 00:55:34.382286 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.382228    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" event={"ID":"e049c769-a863-4f07-8100-3d5d62c681d3","Type":"ContainerStarted","Data":"9c430a27276e94e96c56a730efc69c7e73e87d4c4e59df6b8048dfab9741cb42"}
Nov 03 00:55:34.442098 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:55:34.442066    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc532fe_49e8_404d_ab30_fd29bae9f031.slice/crio-a56762cb16f733882dadc0afdd667ffc73e6a97734fa60c3b840e7e0a65570f7 WatchSource:0}: Error finding container a56762cb16f733882dadc0afdd667ffc73e6a97734fa60c3b840e7e0a65570f7: Status 404 returned error can't find the container with id a56762cb16f733882dadc0afdd667ffc73e6a97734fa60c3b840e7e0a65570f7
Nov 03 00:55:34.445880 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.445851    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"]
Nov 03 00:55:34.647982 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.647956    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:55:34.700527 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.700474    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:55:34.771483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.771451    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-plugins-config-volume\") pod \"cac94ec3-4fa0-4830-b58d-2e304452c921\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") "
Nov 03 00:55:34.771649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.771590    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-provider-credentials\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-provider-credentials\") pod \"cac94ec3-4fa0-4830-b58d-2e304452c921\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") "
Nov 03 00:55:34.771649 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.771640    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-kube-api-access-7sw6j\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-kube-api-access-7sw6j\") pod \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") "
Nov 03 00:55:34.771815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.771775    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-provider-credentials\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-provider-credentials\") pod \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") "
Nov 03 00:55:34.771883 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.771868    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-kube-api-access-fvjx6\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-kube-api-access-fvjx6\") pod \"cac94ec3-4fa0-4830-b58d-2e304452c921\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") "
Nov 03 00:55:34.772129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.771945    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-routing-overlay-volume\") pod \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") "
Nov 03 00:55:34.772129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.772020    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-routing-overlay-volume\") pod \"cac94ec3-4fa0-4830-b58d-2e304452c921\" (UID: \"cac94ec3-4fa0-4830-b58d-2e304452c921\") "
Nov 03 00:55:34.772129 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.772040    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-plugins-config-volume" pod "cac94ec3-4fa0-4830-b58d-2e304452c921" (UID: "cac94ec3-4fa0-4830-b58d-2e304452c921"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:34.772486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.772176    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-plugins-config-volume\") pod \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\" (UID: \"64dd5195-46b0-4e2d-9960-7d6a6026aa04\") "
Nov 03 00:55:34.772955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.772931    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:34.773981 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.773953    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-plugins-config-volume" pod "64dd5195-46b0-4e2d-9960-7d6a6026aa04" (UID: "64dd5195-46b0-4e2d-9960-7d6a6026aa04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:34.774507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.774467    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-routing-overlay-volume" pod "64dd5195-46b0-4e2d-9960-7d6a6026aa04" (UID: "64dd5195-46b0-4e2d-9960-7d6a6026aa04"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:34.774928 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.774881    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-routing-overlay-volume" pod "cac94ec3-4fa0-4830-b58d-2e304452c921" (UID: "cac94ec3-4fa0-4830-b58d-2e304452c921"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:55:34.776323 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.776296    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-provider-credentials" pod "cac94ec3-4fa0-4830-b58d-2e304452c921" (UID: "cac94ec3-4fa0-4830-b58d-2e304452c921"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:34.776847 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.776807    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-provider-credentials" pod "64dd5195-46b0-4e2d-9960-7d6a6026aa04" (UID: "64dd5195-46b0-4e2d-9960-7d6a6026aa04"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:34.778704 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.778678    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-kube-api-access-fvjx6" pod "cac94ec3-4fa0-4830-b58d-2e304452c921" (UID: "cac94ec3-4fa0-4830-b58d-2e304452c921"). InnerVolumeSpecName "kube-api-access-fvjx6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:34.779596 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.779575    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-kube-api-access-7sw6j" pod "64dd5195-46b0-4e2d-9960-7d6a6026aa04" (UID: "64dd5195-46b0-4e2d-9960-7d6a6026aa04"). InnerVolumeSpecName "kube-api-access-7sw6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:55:34.888358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.878409    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/cac94ec3-4fa0-4830-b58d-2e304452c921-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:34.888358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.878449    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:34.888358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.878463    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:34.888358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.878477    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sw6j\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-kube-api-access-7sw6j\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:34.888358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.878490    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/64dd5195-46b0-4e2d-9960-7d6a6026aa04-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:34.888358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.878506    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvjx6\" (UniqueName: \"kubernetes.io/projected/cac94ec3-4fa0-4830-b58d-2e304452c921-kube-api-access-fvjx6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:34.888358 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:34.878521    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/64dd5195-46b0-4e2d-9960-7d6a6026aa04-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:55:35.529836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.529796    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d" event={"ID":"3fc532fe-49e8-404d-ab30-fd29bae9f031","Type":"ContainerStarted","Data":"3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4"}
Nov 03 00:55:35.552677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.552644    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:55:35.552935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.552914    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d" event={"ID":"3fc532fe-49e8-404d-ab30-fd29bae9f031","Type":"ContainerStarted","Data":"a56762cb16f733882dadc0afdd667ffc73e6a97734fa60c3b840e7e0a65570f7"}
Nov 03 00:55:35.553086 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.553073    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:35.568276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.565962    6000 generic.go:356] "Generic (PLEG): container finished" podID="cac94ec3-4fa0-4830-b58d-2e304452c921" containerID="a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4" exitCode=0
Nov 03 00:55:35.568276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.566075    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7" event={"ID":"cac94ec3-4fa0-4830-b58d-2e304452c921","Type":"ContainerDied","Data":"a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4"}
Nov 03 00:55:35.568276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.566125    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7" event={"ID":"cac94ec3-4fa0-4830-b58d-2e304452c921","Type":"ContainerDied","Data":"661c10e14d315833b969aac251cfeb4e490731f9b5cf51ab79a22dc6fb532553"}
Nov 03 00:55:35.568276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.566160    6000 scope.go:122] "RemoveContainer" containerID="a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4"
Nov 03 00:55:35.568276 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.566409    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"
Nov 03 00:55:35.582275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.580110    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" event={"ID":"e049c769-a863-4f07-8100-3d5d62c681d3","Type":"ContainerStarted","Data":"5b347d3bf78d176b532b4c8f11e6cedfabc9d55a23e18f810fae7e7fec98bb8f"}
Nov 03 00:55:35.582275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.581978    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:35.615425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.607927    6000 generic.go:356] "Generic (PLEG): container finished" podID="64dd5195-46b0-4e2d-9960-7d6a6026aa04" containerID="8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889" exitCode=0
Nov 03 00:55:35.615425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.608009    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf" event={"ID":"64dd5195-46b0-4e2d-9960-7d6a6026aa04","Type":"ContainerDied","Data":"8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889"}
Nov 03 00:55:35.615425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.608080    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf" event={"ID":"64dd5195-46b0-4e2d-9960-7d6a6026aa04","Type":"ContainerDied","Data":"f7c48ecce0204d5b2b834e4fbe8c3d371ffb8cee86c4fa839541e65a0c82a729"}
Nov 03 00:55:35.615425 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.608186    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"
Nov 03 00:55:35.621896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.621843    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d" podStartSLOduration=2.621823163 podStartE2EDuration="2.621823163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:35.584157706 +0000 UTC m=+2973.139509730" watchObservedRunningTime="2026-11-03 00:55:35.621823163 +0000 UTC m=+2973.177175188"
Nov 03 00:55:35.626410 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.626388    6000 scope.go:122] "RemoveContainer" containerID="a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4"
Nov 03 00:55:35.626751 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:35.626734    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4\": container with ID starting with a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4 not found: ID does not exist" containerID="a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4"
Nov 03 00:55:35.626811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.626759    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4"} err="failed to get container status \"a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4\": rpc error: code = NotFound desc = could not find container \"a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4\": container with ID starting with a894848ef009c5eb3f1181bed19c27f84f245ea72bba9bd8fcc3a7dfbd9945c4 not found: ID does not exist"
Nov 03 00:55:35.626811 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.626776    6000 scope.go:122] "RemoveContainer" containerID="8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889"
Nov 03 00:55:35.638782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.638729    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" podStartSLOduration=2.638711529 podStartE2EDuration="2.638711529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:35.632844407 +0000 UTC m=+2973.188196431" watchObservedRunningTime="2026-11-03 00:55:35.638711529 +0000 UTC m=+2973.194063554"
Nov 03 00:55:35.639176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.639155    6000 scope.go:122] "RemoveContainer" containerID="8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889"
Nov 03 00:55:35.639715 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:35.639683    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889\": container with ID starting with 8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889 not found: ID does not exist" containerID="8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889"
Nov 03 00:55:35.639849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.639725    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889"} err="failed to get container status \"8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889\": rpc error: code = NotFound desc = could not find container \"8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889\": container with ID starting with 8f5774e70c32e7986af24e1fb9444aa697eaeeb21b4227e60a6412ebe46ca889 not found: ID does not exist"
Nov 03 00:55:35.694731 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.694691    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"]
Nov 03 00:55:35.694935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.694752    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-68bdj9jzf"]
Nov 03 00:55:35.717913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.717886    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"]
Nov 03 00:55:35.725647 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:35.725619    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcq2vv7"]
Nov 03 00:55:36.570892 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:36.570845    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64dd5195-46b0-4e2d-9960-7d6a6026aa04" path="/var/lib/kubelet/pods/64dd5195-46b0-4e2d-9960-7d6a6026aa04/volumes"
Nov 03 00:55:36.571709 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:36.571687    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cac94ec3-4fa0-4830-b58d-2e304452c921" path="/var/lib/kubelet/pods/cac94ec3-4fa0-4830-b58d-2e304452c921/volumes"
Nov 03 00:55:46.400048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.399963    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"]
Nov 03 00:55:46.401249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.401225    6000 cpu_manager.go:414] "containerMap: removing container" podUID="cac94ec3-4fa0-4830-b58d-2e304452c921" containerName="payload-processing"
Nov 03 00:55:46.401249 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.401265    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cac94ec3-4fa0-4830-b58d-2e304452c921" containerName="payload-processing"
Nov 03 00:55:46.401502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.401311    6000 cpu_manager.go:414] "containerMap: removing container" podUID="64dd5195-46b0-4e2d-9960-7d6a6026aa04" containerName="payload-processing"
Nov 03 00:55:46.401502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.401319    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64dd5195-46b0-4e2d-9960-7d6a6026aa04" containerName="payload-processing"
Nov 03 00:55:46.401611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.401525    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="64dd5195-46b0-4e2d-9960-7d6a6026aa04" containerName="payload-processing"
Nov 03 00:55:46.401611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.401560    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="cac94ec3-4fa0-4830-b58d-2e304452c921" containerName="payload-processing"
Nov 03 00:55:46.407705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.407688    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.429796 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.429762    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"]
Nov 03 00:55:46.432560 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.432527    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-84f605\"/\"kube-root-ca.crt\""
Nov 03 00:55:46.434595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.433063    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-84f605\"/\"default-dockercfg-7kv2w\""
Nov 03 00:55:46.434595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.433162    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-84f605\"/\"test-model-84f605-kserve-self-signed-certs\""
Nov 03 00:55:46.434595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.433211    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-84f605\"/\"openshift-service-ca.crt\""
Nov 03 00:55:46.493852 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.493814    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwcrw\" (UniqueName: \"kubernetes.io/projected/9163c109-b64a-4e78-943f-c8e73a20682b-kube-api-access-pwcrw\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.494031 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.494012    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9163c109-b64a-4e78-943f-c8e73a20682b-tls-certs\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.494123 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.494112    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-tmp-dir\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.494216 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.494203    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-dshm\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.494353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.494333    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-model-cache\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.494408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.494395    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-home\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595451 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595420    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-model-cache\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595495    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-home\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595662 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595548    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwcrw\" (UniqueName: \"kubernetes.io/projected/9163c109-b64a-4e78-943f-c8e73a20682b-kube-api-access-pwcrw\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595700    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9163c109-b64a-4e78-943f-c8e73a20682b-tls-certs\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595808    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-tmp-dir\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595834    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-model-cache\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595895    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-home\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.595999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.595987    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-dshm\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.596111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.596091    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-tmp-dir\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.598235 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.598208    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-dshm\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.601354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.601320    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9163c109-b64a-4e78-943f-c8e73a20682b-tls-certs\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.649278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.649161    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwcrw\" (UniqueName: \"kubernetes.io/projected/9163c109-b64a-4e78-943f-c8e73a20682b-kube-api-access-pwcrw\") pod \"test-model-84f605-kserve-6b8d595889-vcj6k\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") " pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:46.733461 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:46.729593    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:47.105674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:47.105646    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"]
Nov 03 00:55:47.105855 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:55:47.105836    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9163c109_b64a_4e78_943f_c8e73a20682b.slice/crio-4e5b74a6c9693c19e8bb1a937063cbabea4dfa0ba12b5c9b750bde49382b4739 WatchSource:0}: Error finding container 4e5b74a6c9693c19e8bb1a937063cbabea4dfa0ba12b5c9b750bde49382b4739: Status 404 returned error can't find the container with id 4e5b74a6c9693c19e8bb1a937063cbabea4dfa0ba12b5c9b750bde49382b4739
Nov 03 00:55:47.724383 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:47.724350    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 00:55:47.724793 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:47.724420    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 00:55:47.833490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:47.832905    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k" event={"ID":"9163c109-b64a-4e78-943f-c8e73a20682b","Type":"ContainerStarted","Data":"8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3"}
Nov 03 00:55:47.833490 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:47.832960    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k" event={"ID":"9163c109-b64a-4e78-943f-c8e73a20682b","Type":"ContainerStarted","Data":"4e5b74a6c9693c19e8bb1a937063cbabea4dfa0ba12b5c9b750bde49382b4739"}
Nov 03 00:55:47.873351 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:47.873297    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k" podStartSLOduration=1.873281983 podStartE2EDuration="1.873281983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:55:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:55:47.872178775 +0000 UTC m=+2985.427530803" watchObservedRunningTime="2026-11-03 00:55:47.873281983 +0000 UTC m=+2985.428634006"
Nov 03 00:55:48.730352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:48.730319    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:49.898406 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:55:49.898369    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:55:56.730538 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:56.730507    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:56.743379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:56.743356    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:55:56.917663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:55:56.917634    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 00:56:11.102528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:11.102495    6000 generic.go:356] "Generic (PLEG): container finished" podID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerID="3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7" exitCode=0
Nov 03 00:56:11.103003 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:11.102570    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" event={"ID":"98fda859-71bf-4b39-bb5c-5a26bd7ef43a","Type":"ContainerDied","Data":"3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7"}
Nov 03 00:56:11.103050 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:11.103010    6000 scope.go:122] "RemoveContainer" containerID="3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7"
Nov 03 00:56:11.104836 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:11.104807    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:56:11.104899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:11.104877    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:56:11.104936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:11.104914    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:56:12.114352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:12.114314    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" event={"ID":"98fda859-71bf-4b39-bb5c-5a26bd7ef43a","Type":"ContainerStarted","Data":"1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef"}
Nov 03 00:56:12.652997 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:12.652966    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:56:12.653182 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:12.653128    6000 patch_prober.go:29] interesting pod/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Nov 03 00:56:12.653230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:12.653179    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Nov 03 00:56:13.653393 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:13.653356    6000 patch_prober.go:29] interesting pod/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Nov 03 00:56:13.653897 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:13.653417    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Nov 03 00:56:14.653091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:14.653055    6000 patch_prober.go:29] interesting pod/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Nov 03 00:56:14.653322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:14.653124    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Nov 03 00:56:15.653023 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:15.652986    6000 patch_prober.go:29] interesting pod/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Nov 03 00:56:15.653452 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:15.653063    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Nov 03 00:56:16.652726 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:16.652694    6000 patch_prober.go:29] interesting pod/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Nov 03 00:56:16.652979 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:16.652959    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Nov 03 00:56:17.652791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:17.652750    6000 patch_prober.go:29] interesting pod/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused" start-of-body=
Nov 03 00:56:17.653409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:17.652812    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.250:15021/healthz/ready\": dial tcp 10.134.0.250:15021: connect: connection refused"
Nov 03 00:56:18.659414 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:18.659381    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:56:18.659917 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:18.659870    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:56:18.660724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:18.660696    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 00:56:19.897460 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:56:19.897421    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:56:35.312719 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:35.312688    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:56:46.466899 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.465524    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"]
Nov 03 00:56:46.483210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.483171    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:46.489678 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.489648    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"]
Nov 03 00:56:46.499002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.498975    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-056a96\""
Nov 03 00:56:46.510972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.510506    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-056a96-dockercfg-jfjcp\""
Nov 03 00:56:46.573624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.573591    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:46.573825 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.573757    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwjbb\" (UniqueName: \"kubernetes.io/projected/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-kube-api-access-fwjbb\") pod \"payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:46.675873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.675835    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:46.676084 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.675943    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwjbb\" (UniqueName: \"kubernetes.io/projected/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-kube-api-access-fwjbb\") pod \"payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:46.676607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.676580    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:46.695041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.695017    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwjbb\" (UniqueName: \"kubernetes.io/projected/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-kube-api-access-fwjbb\") pod \"payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") " pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:46.821020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:46.820947    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:47.132954 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:56:47.132920    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ce6bc1e_3ff5_4d18_b61e_0ac0243c55b4.slice/crio-563431d7c9381c0ef633966ed8da6c3d22c281c75e4212f493637db231764465 WatchSource:0}: Error finding container 563431d7c9381c0ef633966ed8da6c3d22c281c75e4212f493637db231764465: Status 404 returned error can't find the container with id 563431d7c9381c0ef633966ed8da6c3d22c281c75e4212f493637db231764465
Nov 03 00:56:47.133660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:47.133635    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"]
Nov 03 00:56:47.542269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:47.542165    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" event={"ID":"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4","Type":"ContainerStarted","Data":"7086da36cdd72eb4668151c741570d942b43e7960cbbeb8d3bfa771df9a805e6"}
Nov 03 00:56:47.542269 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:47.542224    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" event={"ID":"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4","Type":"ContainerStarted","Data":"563431d7c9381c0ef633966ed8da6c3d22c281c75e4212f493637db231764465"}
Nov 03 00:56:47.542715 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:47.542296    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:56:47.582717 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:47.582660    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" podStartSLOduration=1.5826465870000002 podStartE2EDuration="1.582646587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:56:47.580297198 +0000 UTC m=+3045.135649224" watchObservedRunningTime="2026-11-03 00:56:47.582646587 +0000 UTC m=+3045.137998610"
Nov 03 00:56:49.897715 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:56:49.897681    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:56:58.553154 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:56:58.553112    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 00:57:03.735803 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.735770    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"]
Nov 03 00:57:03.742197 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.742170    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.785723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.785689    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"]
Nov 03 00:57:03.856300 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.856243    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.856705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.856682    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.856867 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.856836    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.857025 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.856875    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q756\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-kube-api-access-6q756\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.958674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.958630    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.958674 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.958674    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.958938 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.958699    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q756\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-kube-api-access-6q756\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.958995 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.958979    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.959545 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.959522    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.959861 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.959842    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.961405 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.961384    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:03.977375 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:03.977351    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q756\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-kube-api-access-6q756\") pod \"payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") " pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:04.060950 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:04.060588    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:04.813311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:04.813205    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"]
Nov 03 00:57:04.823870 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:04.823839    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.004282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.003880    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.004282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.003930    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhjbr\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-kube-api-access-bhjbr\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.004282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.003983    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.004282 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.004111    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.064479 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.064389    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"]
Nov 03 00:57:05.071345 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.071245    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"]
Nov 03 00:57:05.135875 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.135836    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.136206 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.136186    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.136321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.136225    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhjbr\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-kube-api-access-bhjbr\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.136321 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.136292    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.136660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.136636    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.136974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.136952    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.139078 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.138996    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.155140 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.155116    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhjbr\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-kube-api-access-bhjbr\") pod \"payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") " pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.359279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.356525    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:05.715594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.715503    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"]
Nov 03 00:57:05.804532 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.803897    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:05.806939 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.806904    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"]
Nov 03 00:57:05.820701 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.820664    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx" event={"ID":"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7","Type":"ContainerStarted","Data":"f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe"}
Nov 03 00:57:05.821049 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.820730    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx" event={"ID":"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7","Type":"ContainerStarted","Data":"24425d667e4a30320e7f2017e92a665515e712a085bdda6f0325d1c00807d98c"}
Nov 03 00:57:05.822409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.822385    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:05.823504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.823482    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"]
Nov 03 00:57:05.884364 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.884308    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx" podStartSLOduration=2.884289731 podStartE2EDuration="2.884289731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:05.881208126 +0000 UTC m=+3063.436560162" watchObservedRunningTime="2026-11-03 00:57:05.884289731 +0000 UTC m=+3063.439641757"
Nov 03 00:57:05.899868 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.899844    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:05.900015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.899991    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fbrkx\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-kube-api-access-fbrkx\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:05.900091 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.900072    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:05.900159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:05.900129    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.002211 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.002113    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.002510 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.002489    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.002601 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.002565    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fbrkx\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-kube-api-access-fbrkx\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.002663 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.002618    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.003094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.003049    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-plugins-config-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.003346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.003324    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-routing-overlay-volume\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.005114 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.005094    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-provider-credentials\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.021528 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.021504    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fbrkx\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-kube-api-access-fbrkx\") pod \"payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") " pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.130549 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.130521    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.440052 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:57:06.440022    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbee3448_debc_4509_acf1_b2c9f79f983d.slice/crio-ee270d28502f509ca10e17ecb6f6f619fc6870e77b9354157694e50103296faa WatchSource:0}: Error finding container ee270d28502f509ca10e17ecb6f6f619fc6870e77b9354157694e50103296faa: Status 404 returned error can't find the container with id ee270d28502f509ca10e17ecb6f6f619fc6870e77b9354157694e50103296faa
Nov 03 00:57:06.440478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.440455    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"]
Nov 03 00:57:06.831380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.831287    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2" event={"ID":"86725bd9-4289-46ea-9849-068886463c85","Type":"ContainerStarted","Data":"cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730"}
Nov 03 00:57:06.831380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.831343    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2" event={"ID":"86725bd9-4289-46ea-9849-068886463c85","Type":"ContainerStarted","Data":"29f8cd264529eb4af609554483024a9e02985294fc9ad9111a398b2943c1bd11"}
Nov 03 00:57:06.831885 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.831393    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:06.833189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.833165    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x" event={"ID":"fbee3448-debc-4509-acf1-b2c9f79f983d","Type":"ContainerStarted","Data":"463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70"}
Nov 03 00:57:06.833346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.833204    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x" event={"ID":"fbee3448-debc-4509-acf1-b2c9f79f983d","Type":"ContainerStarted","Data":"ee270d28502f509ca10e17ecb6f6f619fc6870e77b9354157694e50103296faa"}
Nov 03 00:57:06.833346 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.833235    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:06.868967 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.868922    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2" podStartSLOduration=2.86890968 podStartE2EDuration="2.86890968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:06.867333241 +0000 UTC m=+3064.422685268" watchObservedRunningTime="2026-11-03 00:57:06.86890968 +0000 UTC m=+3064.424261705"
Nov 03 00:57:06.903544 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:06.903499    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x" podStartSLOduration=1.903485425 podStartE2EDuration="1.903485425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:06.903110389 +0000 UTC m=+3064.458462432" watchObservedRunningTime="2026-11-03 00:57:06.903485425 +0000 UTC m=+3064.458837450"
Nov 03 00:57:17.845602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:17.845515    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:17.845602 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:17.845600    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:17.846152 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:17.845645    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:17.916851 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:17.916808    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"]
Nov 03 00:57:17.952637 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:17.952451    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"]
Nov 03 00:57:17.997634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:17.997598    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"]
Nov 03 00:57:20.008281 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:20.002832    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:57:20.843183 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:20.843149    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"]
Nov 03 00:57:21.472357 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:21.472313    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"]
Nov 03 00:57:21.472963 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:21.472556    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy" containerID="cri-o://9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd" gracePeriod=30
Nov 03 00:57:22.918189 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:22.918139    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2" podUID="922e6643-1a84-4c9d-bf5c-0c10c3a440ba" containerName="payload-processing" containerID="cri-o://cce1a11dfb46631d9afdf8c3335d3f7c95e7608ff401bfae2803b52dc0b16648" gracePeriod=25
Nov 03 00:57:22.953179 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:22.953130    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9" podUID="629dfba3-dec0-43fe-93e4-665f81e14050" containerName="payload-processing" containerID="cri-o://af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e" gracePeriod=25
Nov 03 00:57:22.998733 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:22.998687    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x" podUID="aaaf801b-3355-49fd-84b4-742538542504" containerName="payload-processing" containerID="cri-o://3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b" gracePeriod=25
Nov 03 00:57:23.055837 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.055806    6000 generic.go:356] "Generic (PLEG): container finished" podID="922e6643-1a84-4c9d-bf5c-0c10c3a440ba" containerID="cce1a11dfb46631d9afdf8c3335d3f7c95e7608ff401bfae2803b52dc0b16648" exitCode=0
Nov 03 00:57:23.055986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.055884    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2" event={"ID":"922e6643-1a84-4c9d-bf5c-0c10c3a440ba","Type":"ContainerDied","Data":"cce1a11dfb46631d9afdf8c3335d3f7c95e7608ff401bfae2803b52dc0b16648"}
Nov 03 00:57:23.318166 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.317822    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:57:23.328677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.328654    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:57:23.350631 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.350605    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:57:23.399117 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399069    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-provider-credentials\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-provider-credentials\") pod \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") "
Nov 03 00:57:23.399347 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399313    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-routing-overlay-volume\") pod \"aaaf801b-3355-49fd-84b4-742538542504\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") "
Nov 03 00:57:23.399433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399352    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-plugins-config-volume\") pod \"aaaf801b-3355-49fd-84b4-742538542504\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") "
Nov 03 00:57:23.399433 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399409    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-provider-credentials\") pod \"aaaf801b-3355-49fd-84b4-742538542504\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") "
Nov 03 00:57:23.399750 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399725    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-routing-overlay-volume" pod "aaaf801b-3355-49fd-84b4-742538542504" (UID: "aaaf801b-3355-49fd-84b4-742538542504"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:23.399903 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399874    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-plugins-config-volume" pod "aaaf801b-3355-49fd-84b4-742538542504" (UID: "aaaf801b-3355-49fd-84b4-742538542504"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:23.399999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399880    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-plugins-config-volume\") pod \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") "
Nov 03 00:57:23.399999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.399982    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-kube-api-access-d974l\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-kube-api-access-d974l\") pod \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") "
Nov 03 00:57:23.400485 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.400355    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-plugins-config-volume" pod "922e6643-1a84-4c9d-bf5c-0c10c3a440ba" (UID: "922e6643-1a84-4c9d-bf5c-0c10c3a440ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:23.401226 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.400822    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-kube-api-access-jxfr9\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-kube-api-access-jxfr9\") pod \"aaaf801b-3355-49fd-84b4-742538542504\" (UID: \"aaaf801b-3355-49fd-84b4-742538542504\") "
Nov 03 00:57:23.401226 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.400920    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-routing-overlay-volume\") pod \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\" (UID: \"922e6643-1a84-4c9d-bf5c-0c10c3a440ba\") "
Nov 03 00:57:23.401708 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.401682    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-routing-overlay-volume" pod "922e6643-1a84-4c9d-bf5c-0c10c3a440ba" (UID: "922e6643-1a84-4c9d-bf5c-0c10c3a440ba"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:23.401942 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.401918    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.402022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.401948    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.402022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.401962    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaaf801b-3355-49fd-84b4-742538542504-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.402022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.401974    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.402424 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.402397    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-provider-credentials" pod "aaaf801b-3355-49fd-84b4-742538542504" (UID: "aaaf801b-3355-49fd-84b4-742538542504"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:23.402780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.402756    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-provider-credentials" pod "922e6643-1a84-4c9d-bf5c-0c10c3a440ba" (UID: "922e6643-1a84-4c9d-bf5c-0c10c3a440ba"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:23.404223 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.404184    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-kube-api-access-jxfr9" pod "aaaf801b-3355-49fd-84b4-742538542504" (UID: "aaaf801b-3355-49fd-84b4-742538542504"). InnerVolumeSpecName "kube-api-access-jxfr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:23.404379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.404337    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-kube-api-access-d974l" pod "922e6643-1a84-4c9d-bf5c-0c10c3a440ba" (UID: "922e6643-1a84-4c9d-bf5c-0c10c3a440ba"). InnerVolumeSpecName "kube-api-access-d974l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:23.503159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.503062    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-plugins-config-volume\") pod \"629dfba3-dec0-43fe-93e4-665f81e14050\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") "
Nov 03 00:57:23.503401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.503207    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-kube-api-access-vjzrv\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-kube-api-access-vjzrv\") pod \"629dfba3-dec0-43fe-93e4-665f81e14050\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") "
Nov 03 00:57:23.503401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.503294    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-routing-overlay-volume\") pod \"629dfba3-dec0-43fe-93e4-665f81e14050\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") "
Nov 03 00:57:23.503401 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.503376    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-provider-credentials\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-provider-credentials\") pod \"629dfba3-dec0-43fe-93e4-665f81e14050\" (UID: \"629dfba3-dec0-43fe-93e4-665f81e14050\") "
Nov 03 00:57:23.503603 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.503485    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-plugins-config-volume" pod "629dfba3-dec0-43fe-93e4-665f81e14050" (UID: "629dfba3-dec0-43fe-93e4-665f81e14050"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:23.503729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.503703    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-routing-overlay-volume" pod "629dfba3-dec0-43fe-93e4-665f81e14050" (UID: "629dfba3-dec0-43fe-93e4-665f81e14050"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:23.504472 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.504446    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.504600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.504476    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.504600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.504495    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.504600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.504508    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/629dfba3-dec0-43fe-93e4-665f81e14050-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.504600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.504521    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d974l\" (UniqueName: \"kubernetes.io/projected/922e6643-1a84-4c9d-bf5c-0c10c3a440ba-kube-api-access-d974l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.504600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.504534    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxfr9\" (UniqueName: \"kubernetes.io/projected/aaaf801b-3355-49fd-84b4-742538542504-kube-api-access-jxfr9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.505705 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.505669    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-kube-api-access-vjzrv" pod "629dfba3-dec0-43fe-93e4-665f81e14050" (UID: "629dfba3-dec0-43fe-93e4-665f81e14050"). InnerVolumeSpecName "kube-api-access-vjzrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:23.505921 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.505902    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-provider-credentials" pod "629dfba3-dec0-43fe-93e4-665f81e14050" (UID: "629dfba3-dec0-43fe-93e4-665f81e14050"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:23.619949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.619912    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjzrv\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-kube-api-access-vjzrv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:23.619949 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:23.619940    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/629dfba3-dec0-43fe-93e4-665f81e14050-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:24.065132 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.065092    6000 generic.go:356] "Generic (PLEG): container finished" podID="629dfba3-dec0-43fe-93e4-665f81e14050" containerID="af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e" exitCode=0
Nov 03 00:57:24.065710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.065160    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"
Nov 03 00:57:24.065710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.065175    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9" event={"ID":"629dfba3-dec0-43fe-93e4-665f81e14050","Type":"ContainerDied","Data":"af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e"}
Nov 03 00:57:24.065710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.065300    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9" event={"ID":"629dfba3-dec0-43fe-93e4-665f81e14050","Type":"ContainerDied","Data":"539e4a6eb6321f18df9e59e2d13b1d90e6db36d56cf9c14b705e66b792e0a90a"}
Nov 03 00:57:24.065710 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.065335    6000 scope.go:122] "RemoveContainer" containerID="af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e"
Nov 03 00:57:24.067732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.067691    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2" event={"ID":"922e6643-1a84-4c9d-bf5c-0c10c3a440ba","Type":"ContainerDied","Data":"8d77cc31ff7e75750968046284620bdc411fa08e4ee3f96425cd4232693730be"}
Nov 03 00:57:24.067819 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.067804    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"
Nov 03 00:57:24.074336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.074306    6000 generic.go:356] "Generic (PLEG): container finished" podID="aaaf801b-3355-49fd-84b4-742538542504" containerID="3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b" exitCode=0
Nov 03 00:57:24.074486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.074382    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x" event={"ID":"aaaf801b-3355-49fd-84b4-742538542504","Type":"ContainerDied","Data":"3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b"}
Nov 03 00:57:24.074486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.074394    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"
Nov 03 00:57:24.074486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.074445    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x" event={"ID":"aaaf801b-3355-49fd-84b4-742538542504","Type":"ContainerDied","Data":"1bd0806f5b24132f7348889ae73a4b0e168c31e324e94897696951f363a3be3e"}
Nov 03 00:57:24.078972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.078952    6000 scope.go:122] "RemoveContainer" containerID="af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e"
Nov 03 00:57:24.079333 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:24.079302    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e\": container with ID starting with af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e not found: ID does not exist" containerID="af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e"
Nov 03 00:57:24.079462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.079341    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e"} err="failed to get container status \"af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e\": rpc error: code = NotFound desc = could not find container \"af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e\": container with ID starting with af5f908fd087e630f3c8f97e63ea822b140d962f2d741f6e50e42e35100dca0e not found: ID does not exist"
Nov 03 00:57:24.079462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.079363    6000 scope.go:122] "RemoveContainer" containerID="cce1a11dfb46631d9afdf8c3335d3f7c95e7608ff401bfae2803b52dc0b16648"
Nov 03 00:57:24.090036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.090017    6000 scope.go:122] "RemoveContainer" containerID="3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b"
Nov 03 00:57:24.100107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.100088    6000 scope.go:122] "RemoveContainer" containerID="3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b"
Nov 03 00:57:24.100404 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:24.100384    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b\": container with ID starting with 3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b not found: ID does not exist" containerID="3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b"
Nov 03 00:57:24.100466 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.100412    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b"} err="failed to get container status \"3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b\": rpc error: code = NotFound desc = could not find container \"3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b\": container with ID starting with 3679d52514479a24307a979a8ff59ea3c265873b6c792d8e7b6a5a343eaec02b not found: ID does not exist"
Nov 03 00:57:24.144486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.144462    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"]
Nov 03 00:57:24.148278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.148245    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-67fdq8x4x"]
Nov 03 00:57:24.246280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.232676    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"]
Nov 03 00:57:24.265297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.264388    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-75b7tmpc9"]
Nov 03 00:57:24.268514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.268484    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"]
Nov 03 00:57:24.272548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.272523    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-78c6spdr2"]
Nov 03 00:57:24.572006 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.571932    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="629dfba3-dec0-43fe-93e4-665f81e14050" path="/var/lib/kubelet/pods/629dfba3-dec0-43fe-93e4-665f81e14050/volumes"
Nov 03 00:57:24.572529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.572504    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="922e6643-1a84-4c9d-bf5c-0c10c3a440ba" path="/var/lib/kubelet/pods/922e6643-1a84-4c9d-bf5c-0c10c3a440ba/volumes"
Nov 03 00:57:24.572935 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:24.572920    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaaf801b-3355-49fd-84b4-742538542504" path="/var/lib/kubelet/pods/aaaf801b-3355-49fd-84b4-742538542504/volumes"
Nov 03 00:57:25.844218 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:25.844176    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q" podUID="aaad8926-77d9-4ea6-bd34-00afc826fd84" containerName="payload-processing" containerID="cri-o://dcbba6b9cd8f310415e95e9117fa271224059af76659231d6040d818bf04441f" gracePeriod=25
Nov 03 00:57:26.104083 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.104049    6000 generic.go:356] "Generic (PLEG): container finished" podID="aaad8926-77d9-4ea6-bd34-00afc826fd84" containerID="dcbba6b9cd8f310415e95e9117fa271224059af76659231d6040d818bf04441f" exitCode=0
Nov 03 00:57:26.104237 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.104120    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q" event={"ID":"aaad8926-77d9-4ea6-bd34-00afc826fd84","Type":"ContainerDied","Data":"dcbba6b9cd8f310415e95e9117fa271224059af76659231d6040d818bf04441f"}
Nov 03 00:57:26.181711 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.181663    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:57:26.370673 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.370643    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-routing-overlay-volume\") pod \"aaad8926-77d9-4ea6-bd34-00afc826fd84\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") "
Nov 03 00:57:26.370842 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.370815    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-kube-api-access-brrcd\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-kube-api-access-brrcd\") pod \"aaad8926-77d9-4ea6-bd34-00afc826fd84\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") "
Nov 03 00:57:26.370889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.370843    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-plugins-config-volume\") pod \"aaad8926-77d9-4ea6-bd34-00afc826fd84\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") "
Nov 03 00:57:26.370986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.370953    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-provider-credentials\") pod \"aaad8926-77d9-4ea6-bd34-00afc826fd84\" (UID: \"aaad8926-77d9-4ea6-bd34-00afc826fd84\") "
Nov 03 00:57:26.371125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.370995    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-routing-overlay-volume" pod "aaad8926-77d9-4ea6-bd34-00afc826fd84" (UID: "aaad8926-77d9-4ea6-bd34-00afc826fd84"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:26.371311 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.371288    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-plugins-config-volume" pod "aaad8926-77d9-4ea6-bd34-00afc826fd84" (UID: "aaad8926-77d9-4ea6-bd34-00afc826fd84"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:26.371707 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.371690    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:26.371760 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.371711    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/aaad8926-77d9-4ea6-bd34-00afc826fd84-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:26.373350 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.373324    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-kube-api-access-brrcd" pod "aaad8926-77d9-4ea6-bd34-00afc826fd84" (UID: "aaad8926-77d9-4ea6-bd34-00afc826fd84"). InnerVolumeSpecName "kube-api-access-brrcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:26.373459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.373362    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-provider-credentials" pod "aaad8926-77d9-4ea6-bd34-00afc826fd84" (UID: "aaad8926-77d9-4ea6-bd34-00afc826fd84"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:26.423578 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.423548    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"]
Nov 03 00:57:26.424507 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424490    6000 cpu_manager.go:414] "containerMap: removing container" podUID="922e6643-1a84-4c9d-bf5c-0c10c3a440ba" containerName="payload-processing"
Nov 03 00:57:26.424561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424508    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="922e6643-1a84-4c9d-bf5c-0c10c3a440ba" containerName="payload-processing"
Nov 03 00:57:26.424561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424529    6000 cpu_manager.go:414] "containerMap: removing container" podUID="aaad8926-77d9-4ea6-bd34-00afc826fd84" containerName="payload-processing"
Nov 03 00:57:26.424561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424535    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaad8926-77d9-4ea6-bd34-00afc826fd84" containerName="payload-processing"
Nov 03 00:57:26.424561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424550    6000 cpu_manager.go:414] "containerMap: removing container" podUID="629dfba3-dec0-43fe-93e4-665f81e14050" containerName="payload-processing"
Nov 03 00:57:26.424561 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424555    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="629dfba3-dec0-43fe-93e4-665f81e14050" containerName="payload-processing"
Nov 03 00:57:26.424724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424567    6000 cpu_manager.go:414] "containerMap: removing container" podUID="aaaf801b-3355-49fd-84b4-742538542504" containerName="payload-processing"
Nov 03 00:57:26.424724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424572    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaaf801b-3355-49fd-84b4-742538542504" containerName="payload-processing"
Nov 03 00:57:26.424724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424689    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="922e6643-1a84-4c9d-bf5c-0c10c3a440ba" containerName="payload-processing"
Nov 03 00:57:26.424724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424697    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaad8926-77d9-4ea6-bd34-00afc826fd84" containerName="payload-processing"
Nov 03 00:57:26.424724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424707    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="629dfba3-dec0-43fe-93e4-665f81e14050" containerName="payload-processing"
Nov 03 00:57:26.424724 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.424713    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaaf801b-3355-49fd-84b4-742538542504" containerName="payload-processing"
Nov 03 00:57:26.429175 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.429142    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.447499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.447473    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"]
Nov 03 00:57:26.447499 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.447664    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-7f20f7\""
Nov 03 00:57:26.474056 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.474023    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:26.474200 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.474063    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brrcd\" (UniqueName: \"kubernetes.io/projected/aaad8926-77d9-4ea6-bd34-00afc826fd84-kube-api-access-brrcd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:26.498695 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.498090    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"]
Nov 03 00:57:26.504849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.504826    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:26.518318 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.518241    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"]
Nov 03 00:57:26.587278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.584174    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.587278 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.584380    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5m97v\" (UniqueName: \"kubernetes.io/projected/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-kube-api-access-5m97v\") pod \"payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.612279 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.609874    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-7f20f7-dockercfg-bzc4j\""
Nov 03 00:57:26.689684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.689638    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7529e5d-44c6-4282-b229-a977f6377256-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:26.690094 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.690069    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.690336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.690315    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5m97v\" (UniqueName: \"kubernetes.io/projected/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-kube-api-access-5m97v\") pod \"payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.690607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.690590    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8whs\" (UniqueName: \"kubernetes.io/projected/f7529e5d-44c6-4282-b229-a977f6377256-kube-api-access-b8whs\") pod \"payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:26.694409 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.692331    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.721504 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.719853    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5m97v\" (UniqueName: \"kubernetes.io/projected/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-kube-api-access-5m97v\") pod \"payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") " pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.779337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.752714    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:26.800999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.798917    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7529e5d-44c6-4282-b229-a977f6377256-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:26.800999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.799102    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8whs\" (UniqueName: \"kubernetes.io/projected/f7529e5d-44c6-4282-b229-a977f6377256-kube-api-access-b8whs\") pod \"payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:26.800999 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.800280    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7529e5d-44c6-4282-b229-a977f6377256-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:26.877151 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:26.877107    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8whs\" (UniqueName: \"kubernetes.io/projected/f7529e5d-44c6-4282-b229-a977f6377256-kube-api-access-b8whs\") pod \"payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:27.076607 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.076569    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:57:27.130810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.130769    6000 generic.go:356] "Generic (PLEG): container finished" podID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerID="9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd" exitCode=0
Nov 03 00:57:27.131016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.130803    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" event={"ID":"bb302f98-0db5-4b53-8aaf-00d315f1a60f","Type":"ContainerDied","Data":"9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd"}
Nov 03 00:57:27.131016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.130830    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"
Nov 03 00:57:27.131016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.130879    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr" event={"ID":"bb302f98-0db5-4b53-8aaf-00d315f1a60f","Type":"ContainerDied","Data":"9704e0e7b97d6107f5dec4a4b2f35b5840ea6ae1917dd937e5416363ac51b084"}
Nov 03 00:57:27.131016 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.130917    6000 scope.go:122] "RemoveContainer" containerID="9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd"
Nov 03 00:57:27.133849 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.133810    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"
Nov 03 00:57:27.134377 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.134203    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q" event={"ID":"aaad8926-77d9-4ea6-bd34-00afc826fd84","Type":"ContainerDied","Data":"2d2fe80e82ddecaf9849177c4d5bdec303e0f6d719b3f93dbebc9b263a1d35eb"}
Nov 03 00:57:27.145804 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.145781    6000 scope.go:122] "RemoveContainer" containerID="9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd"
Nov 03 00:57:27.146172 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:27.146152    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd\": container with ID starting with 9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd not found: ID does not exist" containerID="9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd"
Nov 03 00:57:27.146328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.146179    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd"} err="failed to get container status \"9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd\": rpc error: code = NotFound desc = could not find container \"9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd\": container with ID starting with 9d9430d3b2579437a9ae6d00df45b30b1ba22841db63939e8567af90f24fabfd not found: ID does not exist"
Nov 03 00:57:27.146328 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.146194    6000 scope.go:122] "RemoveContainer" containerID="dcbba6b9cd8f310415e95e9117fa271224059af76659231d6040d818bf04441f"
Nov 03 00:57:27.147354 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.147316    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:27.187337 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.185826    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"]
Nov 03 00:57:27.195658 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.195119    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-ac338d/payload-processing-external-model-e2e-extmod-b-ac338d-667442b9q"]
Nov 03 00:57:27.208227 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208199    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-podinfo\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208301    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-certs\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208343    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-token\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-token\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208395 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208386    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istiod-ca-cert\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208451    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-socket\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208548    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-credential-socket\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208571    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-envoy\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208613    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-kube-api-access-k458x\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-kube-api-access-k458x\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.208769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.208639    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-data\") pod \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\" (UID: \"bb302f98-0db5-4b53-8aaf-00d315f1a60f\") "
Nov 03 00:57:27.209355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.209057    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-certs" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:27.209355 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.209149    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-socket" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:27.209685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.209665    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.209745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.209679    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-data" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:27.209745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.209690    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.209745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.209701    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-credential-socket" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:27.210308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.210241    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istiod-ca-cert" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:27.211204 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.211177    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-podinfo" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 00:57:27.211940 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.211919    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-kube-api-access-k458x" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "kube-api-access-k458x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:27.212002 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.211954    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-token" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:27.212137 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.212112    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-envoy" pod "bb302f98-0db5-4b53-8aaf-00d315f1a60f" (UID: "bb302f98-0db5-4b53-8aaf-00d315f1a60f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 00:57:27.240826 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.240767    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"]
Nov 03 00:57:27.323833 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.312161    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.324119 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.324088    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.324297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.324280    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k458x\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-kube-api-access-k458x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.324432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.324414    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.324828 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.324799    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.325103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.325087    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.325391 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.325352    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb302f98-0db5-4b53-8aaf-00d315f1a60f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:27.508310 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.503497    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"]
Nov 03 00:57:27.508310 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.507912    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-w8htr"]
Nov 03 00:57:27.752908 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:27.752870    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"]
Nov 03 00:57:28.145919 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.145881    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" event={"ID":"f7529e5d-44c6-4282-b229-a977f6377256","Type":"ContainerStarted","Data":"9169802c62d48b5f6667e6b69b6f025c2133eaffae93f4da676c45305cea3c4d"}
Nov 03 00:57:28.146462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.145945    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:28.146462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.145970    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" event={"ID":"f7529e5d-44c6-4282-b229-a977f6377256","Type":"ContainerStarted","Data":"cd90c0411bea48e65bf942cb64f02c40706e595ec5b497ba611f008c63cf2436"}
Nov 03 00:57:28.149660 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.149627    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg" event={"ID":"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1","Type":"ContainerStarted","Data":"8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae"}
Nov 03 00:57:28.149791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.149669    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg" event={"ID":"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1","Type":"ContainerStarted","Data":"6a2e3cff76abae41eeb5478f16dab321826b62aaa82b8b9892732aa7e1401af4"}
Nov 03 00:57:28.149791 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.149753    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:28.182783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.182720    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" podStartSLOduration=2.182705948 podStartE2EDuration="2.182705948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:28.180929168 +0000 UTC m=+3085.736281193" watchObservedRunningTime="2026-11-03 00:57:28.182705948 +0000 UTC m=+3085.738057971"
Nov 03 00:57:28.216886 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.216833    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg" podStartSLOduration=2.216818237 podStartE2EDuration="2.216818237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:28.216346778 +0000 UTC m=+3085.771698806" watchObservedRunningTime="2026-11-03 00:57:28.216818237 +0000 UTC m=+3085.772170262"
Nov 03 00:57:28.574777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.574688    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaad8926-77d9-4ea6-bd34-00afc826fd84" path="/var/lib/kubelet/pods/aaad8926-77d9-4ea6-bd34-00afc826fd84/volumes"
Nov 03 00:57:28.575331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.575310    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" path="/var/lib/kubelet/pods/bb302f98-0db5-4b53-8aaf-00d315f1a60f/volumes"
Nov 03 00:57:28.705879 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.705846    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"]
Nov 03 00:57:28.969727 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:28.969692    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:57:29.679818 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:29.679785    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"]
Nov 03 00:57:30.533088 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:30.533050    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"]
Nov 03 00:57:33.706782 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:33.706746    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx" podUID="8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" containerName="payload-processing" containerID="cri-o://f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe" gracePeriod=25
Nov 03 00:57:34.017386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.017358    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:34.085655 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.085614    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-routing-overlay-volume\") pod \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") "
Nov 03 00:57:34.085895 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.085878    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-kube-api-access-6q756\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-kube-api-access-6q756\") pod \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") "
Nov 03 00:57:34.085987 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.085917    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-provider-credentials\") pod \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") "
Nov 03 00:57:34.086045 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.086020    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-plugins-config-volume\") pod \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\" (UID: \"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7\") "
Nov 03 00:57:34.086100 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.086045    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-routing-overlay-volume" pod "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" (UID: "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:34.086611 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.086581    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-plugins-config-volume" pod "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" (UID: "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:34.087081 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.087057    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:34.087205 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.087088    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:34.088600 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.088575    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-kube-api-access-6q756" pod "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" (UID: "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7"). InnerVolumeSpecName "kube-api-access-6q756". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:34.088747 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.088732    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-provider-credentials" pod "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" (UID: "8f3367a2-d7b5-4537-9cc0-d5d3af5035c7"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:34.188103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.188073    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q756\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-kube-api-access-6q756\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:34.188103 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.188099    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:34.270640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.270550    6000 generic.go:356] "Generic (PLEG): container finished" podID="8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" containerID="f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe" exitCode=0
Nov 03 00:57:34.270640 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.270596    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"
Nov 03 00:57:34.270896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.270636    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx" event={"ID":"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7","Type":"ContainerDied","Data":"f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe"}
Nov 03 00:57:34.270896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.270694    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx" event={"ID":"8f3367a2-d7b5-4537-9cc0-d5d3af5035c7","Type":"ContainerDied","Data":"24425d667e4a30320e7f2017e92a665515e712a085bdda6f0325d1c00807d98c"}
Nov 03 00:57:34.270896 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.270726    6000 scope.go:122] "RemoveContainer" containerID="f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe"
Nov 03 00:57:34.284756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.284733    6000 scope.go:122] "RemoveContainer" containerID="f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe"
Nov 03 00:57:34.285067 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:34.285043    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe\": container with ID starting with f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe not found: ID does not exist" containerID="f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe"
Nov 03 00:57:34.285125 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.285076    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe"} err="failed to get container status \"f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe\": rpc error: code = NotFound desc = could not find container \"f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe\": container with ID starting with f47a7052de39ea34976c9e2c116517c9613fb08cd0b94d5aac117c3d059af0fe not found: ID does not exist"
Nov 03 00:57:34.321242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.321210    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"]
Nov 03 00:57:34.332777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.332752    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-850433/payload-processing-external-model-e2e-extmod-a-850433-84dcrccsx"]
Nov 03 00:57:34.611356 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.611323    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" path="/var/lib/kubelet/pods/8f3367a2-d7b5-4537-9cc0-d5d3af5035c7/volumes"
Nov 03 00:57:34.680529 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.680486    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2" podUID="86725bd9-4289-46ea-9849-068886463c85" containerName="payload-processing" containerID="cri-o://cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730" gracePeriod=25
Nov 03 00:57:34.985361 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:34.985333    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:35.128042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.127883    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-provider-credentials\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-provider-credentials\") pod \"86725bd9-4289-46ea-9849-068886463c85\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") "
Nov 03 00:57:35.128042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.127969    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-kube-api-access-bhjbr\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-kube-api-access-bhjbr\") pod \"86725bd9-4289-46ea-9849-068886463c85\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") "
Nov 03 00:57:35.128042 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.127994    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-routing-overlay-volume\") pod \"86725bd9-4289-46ea-9849-068886463c85\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") "
Nov 03 00:57:35.128478 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.128120    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-plugins-config-volume\") pod \"86725bd9-4289-46ea-9849-068886463c85\" (UID: \"86725bd9-4289-46ea-9849-068886463c85\") "
Nov 03 00:57:35.134192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.133919    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-routing-overlay-volume" pod "86725bd9-4289-46ea-9849-068886463c85" (UID: "86725bd9-4289-46ea-9849-068886463c85"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:35.134192 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.134134    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-kube-api-access-bhjbr" pod "86725bd9-4289-46ea-9849-068886463c85" (UID: "86725bd9-4289-46ea-9849-068886463c85"). InnerVolumeSpecName "kube-api-access-bhjbr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:35.134595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.134416    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-provider-credentials" pod "86725bd9-4289-46ea-9849-068886463c85" (UID: "86725bd9-4289-46ea-9849-068886463c85"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:35.134929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.134900    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-plugins-config-volume" pod "86725bd9-4289-46ea-9849-068886463c85" (UID: "86725bd9-4289-46ea-9849-068886463c85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:35.231036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.231001    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:35.231036 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.231041    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhjbr\" (UniqueName: \"kubernetes.io/projected/86725bd9-4289-46ea-9849-068886463c85-kube-api-access-bhjbr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:35.231308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.231055    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:35.231308 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.231070    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/86725bd9-4289-46ea-9849-068886463c85-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:35.281040 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.281005    6000 generic.go:356] "Generic (PLEG): container finished" podID="86725bd9-4289-46ea-9849-068886463c85" containerID="cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730" exitCode=0
Nov 03 00:57:35.281268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.281059    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"
Nov 03 00:57:35.281268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.281092    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2" event={"ID":"86725bd9-4289-46ea-9849-068886463c85","Type":"ContainerDied","Data":"cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730"}
Nov 03 00:57:35.281268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.281150    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2" event={"ID":"86725bd9-4289-46ea-9849-068886463c85","Type":"ContainerDied","Data":"29f8cd264529eb4af609554483024a9e02985294fc9ad9111a398b2943c1bd11"}
Nov 03 00:57:35.281268 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.281168    6000 scope.go:122] "RemoveContainer" containerID="cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730"
Nov 03 00:57:35.294945 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.294920    6000 scope.go:122] "RemoveContainer" containerID="cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730"
Nov 03 00:57:35.295269 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:35.295232    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730\": container with ID starting with cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730 not found: ID does not exist" containerID="cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730"
Nov 03 00:57:35.295353 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.295280    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730"} err="failed to get container status \"cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730\": rpc error: code = NotFound desc = could not find container \"cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730\": container with ID starting with cdde29464f8a17cbf9664b4e7add2162e27399f26b6965a003b55a10a685b730 not found: ID does not exist"
Nov 03 00:57:35.332876 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.332847    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"]
Nov 03 00:57:35.345764 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.345734    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-a-d9e067/payload-processing-external-model-e2e-extmod-a-d9e067-7d5dqspt2"]
Nov 03 00:57:35.349888 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.349852    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"]
Nov 03 00:57:35.350968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.350938    6000 cpu_manager.go:414] "containerMap: removing container" podUID="86725bd9-4289-46ea-9849-068886463c85" containerName="payload-processing"
Nov 03 00:57:35.350968 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.350959    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86725bd9-4289-46ea-9849-068886463c85" containerName="payload-processing"
Nov 03 00:57:35.351242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.350983    6000 cpu_manager.go:414] "containerMap: removing container" podUID="8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" containerName="payload-processing"
Nov 03 00:57:35.351242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.350990    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" containerName="payload-processing"
Nov 03 00:57:35.351242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.351032    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy"
Nov 03 00:57:35.351242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.351039    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy"
Nov 03 00:57:35.351586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.351338    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="86725bd9-4289-46ea-9849-068886463c85" containerName="payload-processing"
Nov 03 00:57:35.351586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.351360    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f3367a2-d7b5-4537-9cc0-d5d3af5035c7" containerName="payload-processing"
Nov 03 00:57:35.351586 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.351379    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb302f98-0db5-4b53-8aaf-00d315f1a60f" containerName="istio-proxy"
Nov 03 00:57:35.357360 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.357334    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.372238 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.372210    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-29d09b-openshift-default-dockercfg-fdbzg\""
Nov 03 00:57:35.387855 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.387787    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"]
Nov 03 00:57:35.433483 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433452    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-token\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.433677 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433652    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-credential-socket\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.433790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433710    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-certs\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.433790 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433758    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-podinfo\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.433913 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433868    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-data\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.433972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433921    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-envoy\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.433972 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433937    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istiod-ca-cert\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.434097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.433983    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8jf6\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-kube-api-access-s8jf6\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.434097 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.434040    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-socket\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.533514 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.533471    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x" podUID="fbee3448-debc-4509-acf1-b2c9f79f983d" containerName="payload-processing" containerID="cri-o://463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70" gracePeriod=25
Nov 03 00:57:35.535864 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.535826    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-socket\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.535898    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-token\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536026    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-credential-socket\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536057    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-certs\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536271 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536106    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-podinfo\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536271 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536246    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-data\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536289    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-socket\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536451    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-envoy\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.536486 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536486    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istiod-ca-cert\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.537783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536581    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8jf6\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-kube-api-access-s8jf6\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.537783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536600    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-credential-socket\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.537783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536702    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-data\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.537783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.536886    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-certs\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.537783 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.537636    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istiod-ca-cert\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.539104 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.539083    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-envoy\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.539756 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.539740    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-podinfo\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.561900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.561876    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-token\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.561986 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.561921    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8jf6\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-kube-api-access-s8jf6\") pod \"e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") " pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:35.680869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:35.680831    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:36.147521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.147494    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:36.293190 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.293157    6000 generic.go:356] "Generic (PLEG): container finished" podID="fbee3448-debc-4509-acf1-b2c9f79f983d" containerID="463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70" exitCode=0
Nov 03 00:57:36.293384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.293225    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"
Nov 03 00:57:36.293384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.293242    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x" event={"ID":"fbee3448-debc-4509-acf1-b2c9f79f983d","Type":"ContainerDied","Data":"463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70"}
Nov 03 00:57:36.293384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.293310    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x" event={"ID":"fbee3448-debc-4509-acf1-b2c9f79f983d","Type":"ContainerDied","Data":"ee270d28502f509ca10e17ecb6f6f619fc6870e77b9354157694e50103296faa"}
Nov 03 00:57:36.293384 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.293333    6000 scope.go:122] "RemoveContainer" containerID="463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70"
Nov 03 00:57:36.304348 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.304319    6000 scope.go:122] "RemoveContainer" containerID="463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70"
Nov 03 00:57:36.304672 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:36.304644    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70\": container with ID starting with 463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70 not found: ID does not exist" containerID="463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70"
Nov 03 00:57:36.304758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.304684    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70"} err="failed to get container status \"463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70\": rpc error: code = NotFound desc = could not find container \"463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70\": container with ID starting with 463bb37b783f12112e8d28835ff41f9e0f44bc14a6a709e02d9845b77e194e70 not found: ID does not exist"
Nov 03 00:57:36.316769 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.316750    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-plugins-config-volume\") pod \"fbee3448-debc-4509-acf1-b2c9f79f983d\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") "
Nov 03 00:57:36.317077 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.317059    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-kube-api-access-fbrkx\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-kube-api-access-fbrkx\") pod \"fbee3448-debc-4509-acf1-b2c9f79f983d\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") "
Nov 03 00:57:36.317150 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.317111    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-routing-overlay-volume\") pod \"fbee3448-debc-4509-acf1-b2c9f79f983d\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") "
Nov 03 00:57:36.317220 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.317170    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-plugins-config-volume" pod "fbee3448-debc-4509-acf1-b2c9f79f983d" (UID: "fbee3448-debc-4509-acf1-b2c9f79f983d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:36.317322 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.317216    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-provider-credentials\") pod \"fbee3448-debc-4509-acf1-b2c9f79f983d\" (UID: \"fbee3448-debc-4509-acf1-b2c9f79f983d\") "
Nov 03 00:57:36.317462 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.317442    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-routing-overlay-volume" pod "fbee3448-debc-4509-acf1-b2c9f79f983d" (UID: "fbee3448-debc-4509-acf1-b2c9f79f983d"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 00:57:36.317869 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.317850    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:36.317941 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.317873    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fbee3448-debc-4509-acf1-b2c9f79f983d-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:36.319212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.319192    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-kube-api-access-fbrkx" pod "fbee3448-debc-4509-acf1-b2c9f79f983d" (UID: "fbee3448-debc-4509-acf1-b2c9f79f983d"). InnerVolumeSpecName "kube-api-access-fbrkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:36.319404 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.319384    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-provider-credentials" pod "fbee3448-debc-4509-acf1-b2c9f79f983d" (UID: "fbee3448-debc-4509-acf1-b2c9f79f983d"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 00:57:36.342727 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:57:36.342703    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e692c0_72c9_4292_ae3b_454ff2d5a25f.slice/crio-357296bb81c0f0d8ace7dbc51fb78027e72be6338ae9a9868c6bd55ad5021bb2 WatchSource:0}: Error finding container 357296bb81c0f0d8ace7dbc51fb78027e72be6338ae9a9868c6bd55ad5021bb2: Status 404 returned error can't find the container with id 357296bb81c0f0d8ace7dbc51fb78027e72be6338ae9a9868c6bd55ad5021bb2
Nov 03 00:57:36.345531 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.345507    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"]
Nov 03 00:57:36.345827 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.345800    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:57:36.345877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.345864    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:57:36.345926 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.345896    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:57:36.419634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.419605    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:36.419634 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.419631    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fbrkx\" (UniqueName: \"kubernetes.io/projected/fbee3448-debc-4509-acf1-b2c9f79f983d-kube-api-access-fbrkx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 00:57:36.571812 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.571780    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86725bd9-4289-46ea-9849-068886463c85" path="/var/lib/kubelet/pods/86725bd9-4289-46ea-9849-068886463c85/volumes"
Nov 03 00:57:36.638380 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.638346    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"]
Nov 03 00:57:36.640174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.640144    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-extmod-b-fdacdb/payload-processing-external-model-e2e-extmod-b-fdacdb-694flc57x"]
Nov 03 00:57:36.821969 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.821934    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"]
Nov 03 00:57:36.822918 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.822895    6000 cpu_manager.go:414] "containerMap: removing container" podUID="fbee3448-debc-4509-acf1-b2c9f79f983d" containerName="payload-processing"
Nov 03 00:57:36.822918 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.822914    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbee3448-debc-4509-acf1-b2c9f79f983d" containerName="payload-processing"
Nov 03 00:57:36.823117 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.823086    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbee3448-debc-4509-acf1-b2c9f79f983d" containerName="payload-processing"
Nov 03 00:57:36.827503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.827475    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:36.842362 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.842328    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-056a96-serving-cert\""
Nov 03 00:57:36.846666 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.846640    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"]
Nov 03 00:57:36.927087 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.927011    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:36.927242 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.927090    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x7t4\" (UniqueName: \"kubernetes.io/projected/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-kube-api-access-6x7t4\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:36.927315 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.927284    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:36.927373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:36.927361    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-metrics-certs\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.028525 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.028479    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-metrics-certs\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.028732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.028586    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.028732 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.028673    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x7t4\" (UniqueName: \"kubernetes.io/projected/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-kube-api-access-6x7t4\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.028904 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.028881    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.029272 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.029232    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-openshift-service-ca\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.031212 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.031189    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-metrics-certs\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.031438 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.031418    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-maas-api-tls\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.047936 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.047905    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x7t4\" (UniqueName: \"kubernetes.io/projected/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-kube-api-access-6x7t4\") pod \"maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.169313 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.163958    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:37.305786 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.305743    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" event={"ID":"14e692c0-72c9-4292-ae3b-454ff2d5a25f","Type":"ContainerStarted","Data":"a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d"}
Nov 03 00:57:37.305973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.305837    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" event={"ID":"14e692c0-72c9-4292-ae3b-454ff2d5a25f","Type":"ContainerStarted","Data":"357296bb81c0f0d8ace7dbc51fb78027e72be6338ae9a9868c6bd55ad5021bb2"}
Nov 03 00:57:37.347447 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.347390    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podStartSLOduration=2.347374978 podStartE2EDuration="2.347374978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:37.344626448 +0000 UTC m=+3094.899978478" watchObservedRunningTime="2026-11-03 00:57:37.347374978 +0000 UTC m=+3094.902727001"
Nov 03 00:57:37.516142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.516055    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"]
Nov 03 00:57:37.681467 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.681434    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:37.687918 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:37.687889    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:38.316208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:38.316111    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk" event={"ID":"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f","Type":"ContainerStarted","Data":"537e43e9eb85502508dda77fc152dbdd89be351df2d075ebb7da5de1c6485011"}
Nov 03 00:57:38.316208 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:38.316202    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:38.316843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:38.316222    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk" event={"ID":"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f","Type":"ContainerStarted","Data":"ae2b253d5568bd571766bdebd0f8e9a0d22c4c8aeda86ea4f7d328258d1fee95"}
Nov 03 00:57:38.316843 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:38.316547    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:38.317509 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:38.317485    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:57:38.368702 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:38.368649    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk" podStartSLOduration=2.024240569 podStartE2EDuration="2.368633126s" totalImagesPullingTime="344.392557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:38.362496698 +0000 UTC m=+3095.917848724" watchObservedRunningTime="2026-11-03 00:57:38.368633126 +0000 UTC m=+3095.923985206"
Nov 03 00:57:38.587599 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:38.587496    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbee3448-debc-4509-acf1-b2c9f79f983d" path="/var/lib/kubelet/pods/fbee3448-debc-4509-acf1-b2c9f79f983d/volumes"
Nov 03 00:57:39.181623 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:39.181594    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 00:57:39.181815 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:39.181670    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 00:57:41.722369 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.722337    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"]
Nov 03 00:57:41.729816 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.729790    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:41.752597 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.752572    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-29d09b\""
Nov 03 00:57:41.770830 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.770796    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"]
Nov 03 00:57:41.812240 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.812200    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"]
Nov 03 00:57:41.824834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.824797    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:41.828470 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.828441    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68389007-bcab-4bf9-a05a-dd6e469baefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:41.828650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.828509    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l6kv\" (UniqueName: \"kubernetes.io/projected/82f00888-aaea-4d21-a0f4-dabaedf20c74-kube-api-access-2l6kv\") pod \"payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:41.828650 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.828618    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f00888-aaea-4d21-a0f4-dabaedf20c74-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:41.828820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.828798    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgj2c\" (UniqueName: \"kubernetes.io/projected/68389007-bcab-4bf9-a05a-dd6e469baefe-kube-api-access-vgj2c\") pod \"payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:41.887281 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.884810    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-29d09b-dockercfg-fndvd\""
Nov 03 00:57:41.945565 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.945519    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgj2c\" (UniqueName: \"kubernetes.io/projected/68389007-bcab-4bf9-a05a-dd6e469baefe-kube-api-access-vgj2c\") pod \"payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:41.946022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.945997    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68389007-bcab-4bf9-a05a-dd6e469baefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:41.946214 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.946197    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l6kv\" (UniqueName: \"kubernetes.io/projected/82f00888-aaea-4d21-a0f4-dabaedf20c74-kube-api-access-2l6kv\") pod \"payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:41.987334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.959379    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f00888-aaea-4d21-a0f4-dabaedf20c74-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:41.987334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.947555    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68389007-bcab-4bf9-a05a-dd6e469baefe-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:41.987334 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:41.960489    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f00888-aaea-4d21-a0f4-dabaedf20c74-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:42.016379 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:42.007545    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"]
Nov 03 00:57:42.092696 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:42.082388    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgj2c\" (UniqueName: \"kubernetes.io/projected/68389007-bcab-4bf9-a05a-dd6e469baefe-kube-api-access-vgj2c\") pod \"payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:42.102332 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:42.102035    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l6kv\" (UniqueName: \"kubernetes.io/projected/82f00888-aaea-4d21-a0f4-dabaedf20c74-kube-api-access-2l6kv\") pod \"payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:42.166739 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:42.166702    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:42.348500 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:42.348461    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:42.488065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:42.488033    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"]
Nov 03 00:57:42.488789 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:57:42.488752    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68389007_bcab_4bf9_a05a_dd6e469baefe.slice/crio-90390a70237d463329bf7292d77dd07de4fc8515499cad6655199e28a3ef00b8 WatchSource:0}: Error finding container 90390a70237d463329bf7292d77dd07de4fc8515499cad6655199e28a3ef00b8: Status 404 returned error can't find the container with id 90390a70237d463329bf7292d77dd07de4fc8515499cad6655199e28a3ef00b8
Nov 03 00:57:42.657470 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:57:42.657439    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82f00888_aaea_4d21_a0f4_dabaedf20c74.slice/crio-2abf9b195ee8d8a35221cecf7de3fac748378ba9741f593e1004fe9e636fcd22 WatchSource:0}: Error finding container 2abf9b195ee8d8a35221cecf7de3fac748378ba9741f593e1004fe9e636fcd22: Status 404 returned error can't find the container with id 2abf9b195ee8d8a35221cecf7de3fac748378ba9741f593e1004fe9e636fcd22
Nov 03 00:57:42.657745 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:42.657727    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"]
Nov 03 00:57:43.371638 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.371602    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" event={"ID":"82f00888-aaea-4d21-a0f4-dabaedf20c74","Type":"ContainerStarted","Data":"a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7"}
Nov 03 00:57:43.372162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.371665    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" event={"ID":"82f00888-aaea-4d21-a0f4-dabaedf20c74","Type":"ContainerStarted","Data":"2abf9b195ee8d8a35221cecf7de3fac748378ba9741f593e1004fe9e636fcd22"}
Nov 03 00:57:43.372162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.371700    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:43.373902 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.373878    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" event={"ID":"68389007-bcab-4bf9-a05a-dd6e469baefe","Type":"ContainerStarted","Data":"65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9"}
Nov 03 00:57:43.374022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.373920    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" event={"ID":"68389007-bcab-4bf9-a05a-dd6e469baefe","Type":"ContainerStarted","Data":"90390a70237d463329bf7292d77dd07de4fc8515499cad6655199e28a3ef00b8"}
Nov 03 00:57:43.374022 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.373986    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:43.410580 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.410532    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" podStartSLOduration=2.410517967 podStartE2EDuration="2.410517967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:43.408098072 +0000 UTC m=+3100.963450095" watchObservedRunningTime="2026-11-03 00:57:43.410517967 +0000 UTC m=+3100.965869990"
Nov 03 00:57:43.446503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.446469    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"]
Nov 03 00:57:43.452541 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.452491    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" podStartSLOduration=2.452476812 podStartE2EDuration="2.452476812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:43.447356596 +0000 UTC m=+3101.002708621" watchObservedRunningTime="2026-11-03 00:57:43.452476812 +0000 UTC m=+3101.007828835"
Nov 03 00:57:43.452694 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.452516    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:43.466777 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.466748    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-8ed55a\""
Nov 03 00:57:43.467111 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.467077    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-8ed55a-dockercfg-brz42\""
Nov 03 00:57:43.469974 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.469833    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"]
Nov 03 00:57:43.515624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.515583    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkhdv\" (UniqueName: \"kubernetes.io/projected/45af085b-efbd-4726-8640-8f32e1ee55a8-kube-api-access-jkhdv\") pod \"payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:43.515855 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.515837    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45af085b-efbd-4726-8640-8f32e1ee55a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:43.761336 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.752104    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkhdv\" (UniqueName: \"kubernetes.io/projected/45af085b-efbd-4726-8640-8f32e1ee55a8-kube-api-access-jkhdv\") pod \"payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:43.761900 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.761864    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45af085b-efbd-4726-8640-8f32e1ee55a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:43.776290 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.763467    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45af085b-efbd-4726-8640-8f32e1ee55a8-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:43.841280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:43.836410    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkhdv\" (UniqueName: \"kubernetes.io/projected/45af085b-efbd-4726-8640-8f32e1ee55a8-kube-api-access-jkhdv\") pod \"payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") " pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:44.069503 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:44.069429    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:44.336386 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:44.336306    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 00:57:44.374739 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:57:44.374707    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45af085b_efbd_4726_8640_8f32e1ee55a8.slice/crio-e6b449c0245c4ab60bb1017771e966222d92f7a736ddf895938bcec4e55aeb40 WatchSource:0}: Error finding container e6b449c0245c4ab60bb1017771e966222d92f7a736ddf895938bcec4e55aeb40: Status 404 returned error can't find the container with id e6b449c0245c4ab60bb1017771e966222d92f7a736ddf895938bcec4e55aeb40
Nov 03 00:57:44.375159 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:44.374735    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"]
Nov 03 00:57:45.394753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:45.394712    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" event={"ID":"45af085b-efbd-4726-8640-8f32e1ee55a8","Type":"ContainerStarted","Data":"4878b75c730000b71370a459a96269f67a67b9b65436ca6243238823281834b7"}
Nov 03 00:57:45.395247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:45.394793    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" event={"ID":"45af085b-efbd-4726-8640-8f32e1ee55a8","Type":"ContainerStarted","Data":"e6b449c0245c4ab60bb1017771e966222d92f7a736ddf895938bcec4e55aeb40"}
Nov 03 00:57:45.395247 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:45.394818    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:57:45.432740 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:45.432690    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" podStartSLOduration=2.432676393 podStartE2EDuration="2.432676393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:57:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:57:45.430524478 +0000 UTC m=+3102.985876502" watchObservedRunningTime="2026-11-03 00:57:45.432676393 +0000 UTC m=+3102.988028416"
Nov 03 00:57:49.953640 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:57:49.953594    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:57:54.387955 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:54.387928    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 00:57:54.388477 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:54.388005    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 00:57:56.406758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:57:56.406731    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 00:58:11.697297 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:11.696958    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:58:19.900459 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:58:19.900409    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:58:36.195173 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.195134    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"]
Nov 03 00:58:36.200626 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.200605    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.220807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.220771    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"]
Nov 03 00:58:36.226697 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.226660    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-8ed55a-serving-cert\""
Nov 03 00:58:36.258758 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.258725    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.258964 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.258799    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vpdp\" (UniqueName: \"kubernetes.io/projected/b10511c0-de08-4d50-9be5-976d1b7e0887-kube-api-access-9vpdp\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.259024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.258982    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10511c0-de08-4d50-9be5-976d1b7e0887-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.259065 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.259050    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-metrics-certs\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.360161 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.360130    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-metrics-certs\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.360442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.360311    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.360442 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.360361    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vpdp\" (UniqueName: \"kubernetes.io/projected/b10511c0-de08-4d50-9be5-976d1b7e0887-kube-api-access-9vpdp\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.360588 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:58:36.360450    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-serving-cert: secret "maas-api-e2e-inf-a-8ed55a-serving-cert" not found
Nov 03 00:58:36.360588 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.360466    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10511c0-de08-4d50-9be5-976d1b7e0887-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.360588 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:58:36.360529    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls podName:b10511c0-de08-4d50-9be5-976d1b7e0887 nodeName:}" failed. No retries permitted until 2026-11-03 00:58:36.860508799 +0000 UTC m=+3154.415860801 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls") pod "maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" (UID: "b10511c0-de08-4d50-9be5-976d1b7e0887") : secret "maas-api-e2e-inf-a-8ed55a-serving-cert" not found
Nov 03 00:58:36.361184 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.361159    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10511c0-de08-4d50-9be5-976d1b7e0887-openshift-service-ca\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.362946 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.362923    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-metrics-certs\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.407280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.406321    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vpdp\" (UniqueName: \"kubernetes.io/projected/b10511c0-de08-4d50-9be5-976d1b7e0887-kube-api-access-9vpdp\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.877595 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.877536    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:36.880798 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:36.880758    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls\") pod \"maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:37.117205 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:37.117173    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:37.503331 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:37.499033    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"]
Nov 03 00:58:37.503331 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:58:37.501385    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb10511c0_de08_4d50_9be5_976d1b7e0887.slice/crio-bad3ca36894822c0e7eb58dc5e2f75d0a935cc998f2dc9659b3abb9d1d3488be WatchSource:0}: Error finding container bad3ca36894822c0e7eb58dc5e2f75d0a935cc998f2dc9659b3abb9d1d3488be: Status 404 returned error can't find the container with id bad3ca36894822c0e7eb58dc5e2f75d0a935cc998f2dc9659b3abb9d1d3488be
Nov 03 00:58:37.504550 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:37.504524    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 00:58:37.989540 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:37.989498    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" event={"ID":"b10511c0-de08-4d50-9be5-976d1b7e0887","Type":"ContainerStarted","Data":"bad3ca36894822c0e7eb58dc5e2f75d0a935cc998f2dc9659b3abb9d1d3488be"}
Nov 03 00:58:38.999098 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:38.999055    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" event={"ID":"b10511c0-de08-4d50-9be5-976d1b7e0887","Type":"ContainerStarted","Data":"fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242"}
Nov 03 00:58:38.999569 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:38.999155    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:39.038107 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:39.038057    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" podStartSLOduration=2.654741251 podStartE2EDuration="3.038040047s" totalImagesPullingTime="383.298796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:39.036783791 +0000 UTC m=+3156.592135814" watchObservedRunningTime="2026-11-03 00:58:39.038040047 +0000 UTC m=+3156.593392072"
Nov 03 00:58:45.028048 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.027970    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 00:58:45.364594 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.364560    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"]
Nov 03 00:58:45.370951 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.370922    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.388411 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.388377    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"]
Nov 03 00:58:45.393142 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.393114    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-056a96\"/\"openshift-service-ca.crt\""
Nov 03 00:58:45.393635 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.393603    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-056a96\"/\"kube-root-ca.crt\""
Nov 03 00:58:45.393759 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.393734    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-056a96\"/\"default-dockercfg-whmcb\""
Nov 03 00:58:45.393877 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.393851    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-056a96\"/\"test-model-056a96-kserve-self-signed-certs\""
Nov 03 00:58:45.449502 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.449466    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-dshm\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.449679 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.449526    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-tmp-dir\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.449729 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.449673    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997abdcd-62e3-489c-9150-653c4573d8e3-tls-certs\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.449801 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.449785    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-model-cache\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.449871 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.449856    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-home\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.449973 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.449957    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlkj\" (UniqueName: \"kubernetes.io/projected/997abdcd-62e3-489c-9150-653c4573d8e3-kube-api-access-xxlkj\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.551521 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.551483    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlkj\" (UniqueName: \"kubernetes.io/projected/997abdcd-62e3-489c-9150-653c4573d8e3-kube-api-access-xxlkj\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.551725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.551639    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-dshm\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.551725 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.551680    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-tmp-dir\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.551873 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.551765    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997abdcd-62e3-489c-9150-653c4573d8e3-tls-certs\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.551929 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.551874    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-model-cache\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.552055 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.551928    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-home\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.552141 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.552113    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-tmp-dir\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.552239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.552219    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-model-cache\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.552239 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.552234    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-home\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.554162 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.554136    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-dshm\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.554432 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.554415    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997abdcd-62e3-489c-9150-653c4573d8e3-tls-certs\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.570188 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.570168    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlkj\" (UniqueName: \"kubernetes.io/projected/997abdcd-62e3-489c-9150-653c4573d8e3-kube-api-access-xxlkj\") pod \"test-model-056a96-kserve-75fd754d5-vmxlk\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") " pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:45.693339 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:45.691350    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:46.017713 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:58:46.017671    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod997abdcd_62e3_489c_9150_653c4573d8e3.slice/crio-92627f28949dae19dfe273ce639d4c86be2979d8704f0f8c14acf3c43620ffff WatchSource:0}: Error finding container 92627f28949dae19dfe273ce639d4c86be2979d8704f0f8c14acf3c43620ffff: Status 404 returned error can't find the container with id 92627f28949dae19dfe273ce639d4c86be2979d8704f0f8c14acf3c43620ffff
Nov 03 00:58:46.018872 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:46.018848    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"]
Nov 03 00:58:46.074453 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:46.074423    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk" event={"ID":"997abdcd-62e3-489c-9150-653c4573d8e3","Type":"ContainerStarted","Data":"92627f28949dae19dfe273ce639d4c86be2979d8704f0f8c14acf3c43620ffff"}
Nov 03 00:58:47.085285 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:47.085232    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk" event={"ID":"997abdcd-62e3-489c-9150-653c4573d8e3","Type":"ContainerStarted","Data":"d82674a4de0c0c819e37fab9a219dc1cfb56bbdb9a3a780d7ea6d256ac08996f"}
Nov 03 00:58:47.127659 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:47.127608    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk" podStartSLOduration=2.127594922 podStartE2EDuration="2.127594922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:58:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:58:47.124528378 +0000 UTC m=+3164.679880402" watchObservedRunningTime="2026-11-03 00:58:47.127594922 +0000 UTC m=+3164.682946946"
Nov 03 00:58:47.692624 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:47.692595    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:49.897530 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:58:49.897492    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:58:55.692314 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:55.692281    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:55.705455 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:55.705425    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:58:56.170054 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:58:56.170025    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 00:59:18.178284 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:18.170593    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 00:59:19.906299 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:59:19.905470    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 00:59:30.531810 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:30.531715    6000 generic.go:356] "Generic (PLEG): container finished" podID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerID="a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d" exitCode=0
Nov 03 00:59:30.532970 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:30.532917    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" event={"ID":"14e692c0-72c9-4292-ae3b-454ff2d5a25f","Type":"ContainerDied","Data":"a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d"}
Nov 03 00:59:30.533831 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:30.533806    6000 scope.go:122] "RemoveContainer" containerID="a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d"
Nov 03 00:59:30.536017 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:30.535985    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:59:30.536099 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:30.536060    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:59:30.536157 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:30.536097    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:59:31.542516 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:31.542478    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" event={"ID":"14e692c0-72c9-4292-ae3b-454ff2d5a25f","Type":"ContainerStarted","Data":"107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47"}
Nov 03 00:59:31.681992 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:31.681958    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:59:31.682176 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:31.682124    6000 patch_prober.go:29] interesting pod/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Nov 03 00:59:31.682224 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:31.682170    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Nov 03 00:59:32.681881 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:32.681841    6000 patch_prober.go:29] interesting pod/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Nov 03 00:59:32.682419 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:32.681908    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Nov 03 00:59:33.681784 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:33.681748    6000 patch_prober.go:29] interesting pod/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Nov 03 00:59:33.682041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:33.681806    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Nov 03 00:59:34.681488 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:34.681454    6000 patch_prober.go:29] interesting pod/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Nov 03 00:59:34.681685 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:34.681505    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Nov 03 00:59:35.692280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:35.685621    6000 patch_prober.go:29] interesting pod/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Nov 03 00:59:35.692280 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:35.685696    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Nov 03 00:59:36.681825 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:36.681791    6000 patch_prober.go:29] interesting pod/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body=
Nov 03 00:59:36.682024 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:36.681852    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused"
Nov 03 00:59:37.864840 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:37.864806    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:59:37.865780 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:37.865756    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:59:37.866682 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:37.866658    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 00:59:41.098753 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.098717    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"]
Nov 03 00:59:41.104148 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.104124    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.132584 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.132553    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-056a96-openshift-default-dockercfg-fvmh5\""
Nov 03 00:59:41.137882 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.137855    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"]
Nov 03 00:59:41.220642 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.220602    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-data\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.220821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.220666    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.220821 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.220761    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-token\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.220960 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.220945    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-podinfo\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.221001 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.220993    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istiod-ca-cert\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.221041 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.221017    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-credential-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.221134 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.221119    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9gz8\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-kube-api-access-g9gz8\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.221174 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.221154    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-certs\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.221210 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.221187    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-envoy\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.338834 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.338790    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339051 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.338867    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-token\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339228 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.339206    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-podinfo\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.339304    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istiod-ca-cert\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339408 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.339340    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-credential-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.339442    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9gz8\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-kube-api-access-g9gz8\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.339487    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-certs\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.339511    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-envoy\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.339965 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.339566    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-data\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.340352 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.340115    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-data\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.340423 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.340386    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.353344 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.353037    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istiod-ca-cert\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.353548 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.353465    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-credential-socket\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.353723 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.353700    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-certs\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.360562 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.360530    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-envoy\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.360741 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.360599    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-podinfo\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.362994 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.362971    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-token\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.370230 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.370208    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9gz8\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-kube-api-access-g9gz8\") pod \"e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") " pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.418748 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.418724    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:41.739508 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.739471    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"]
Nov 03 00:59:41.740218 ip-10-0-136-97 kubenswrapper[6000]: W1103 00:59:41.740188    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb63488ad_6f80_48ed_8b9e_66ca54e9fff1.slice/crio-9707f4c4341bdd01e7b85f1f9bfe497142b475467d17a8a472b6f3232cc4e1b5 WatchSource:0}: Error finding container 9707f4c4341bdd01e7b85f1f9bfe497142b475467d17a8a472b6f3232cc4e1b5: Status 404 returned error can't find the container with id 9707f4c4341bdd01e7b85f1f9bfe497142b475467d17a8a472b6f3232cc4e1b5
Nov 03 00:59:41.744684 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.744641    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:59:41.744775 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.744738    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:59:41.744820 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.744800    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 00:59:41.871275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.871179    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" event={"ID":"b63488ad-6f80-48ed-8b9e-66ca54e9fff1","Type":"ContainerStarted","Data":"5c2a594a3a053e7a3ed6c75ce6aac973fbb8919f5ad771c520aff7c5c2997472"}
Nov 03 00:59:41.871275 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.871249    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" event={"ID":"b63488ad-6f80-48ed-8b9e-66ca54e9fff1","Type":"ContainerStarted","Data":"9707f4c4341bdd01e7b85f1f9bfe497142b475467d17a8a472b6f3232cc4e1b5"}
Nov 03 00:59:41.914373 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:41.914318    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podStartSLOduration=0.914303275 podStartE2EDuration="914.303275ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 00:59:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 00:59:41.910547772 +0000 UTC m=+3219.465899797" watchObservedRunningTime="2026-11-03 00:59:41.914303275 +0000 UTC m=+3219.469655300"
Nov 03 00:59:42.419570 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:42.419539    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:42.421657 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:42.421628    6000 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Nov 03 00:59:42.421807 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:42.421682    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Nov 03 00:59:43.420014 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:43.419975    6000 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Nov 03 00:59:43.420459 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:43.420027    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Nov 03 00:59:44.419802 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:44.419758    6000 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Nov 03 00:59:44.420020 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:44.419829    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Nov 03 00:59:45.428865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:45.427980    6000 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Nov 03 00:59:45.428865 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:45.428047    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Nov 03 00:59:46.419767 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:46.419732    6000 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Nov 03 00:59:46.419957 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:46.419784    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Nov 03 00:59:47.419794 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:47.419758    6000 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Nov 03 00:59:47.420338 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:47.419820    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Nov 03 00:59:48.420038 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:48.419998    6000 patch_prober.go:29] interesting pod/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused" start-of-body=
Nov 03 00:59:48.420576 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:48.420078    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.17:15021/healthz/ready\": dial tcp 10.134.1.17:15021: connect: connection refused"
Nov 03 00:59:49.425312 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:49.425275    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:49.425889 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:49.425761    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:49.427015 ip-10-0-136-97 kubenswrapper[6000]: I1103 00:59:49.426992    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 00:59:49.912114 ip-10-0-136-97 kubenswrapper[6000]: E1103 00:59:49.912078    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:00:00.324368 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.324328    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"]
Nov 03 01:00:00.332273 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.332229    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"]
Nov 03 01:00:00.332524 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.332479    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.342031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.342006    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"]
Nov 03 01:00:00.342373 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.342121    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.354711 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.354678    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"]
Nov 03 01:00:00.355238 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.354800    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.364855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.359625    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"]
Nov 03 01:00:00.364855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.360883    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:00:00.366563 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.366543    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"]
Nov 03 01:00:00.379481 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.379456    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"]
Nov 03 01:00:00.379627 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.379570    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:00.424931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.415660    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b7b142-012c-4838-81a7-2404a362caa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.424931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.415726    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lntq4\" (UniqueName: \"kubernetes.io/projected/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-kube-api-access-lntq4\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.424931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.416652    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.424931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.416699    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29nz7\" (UniqueName: \"kubernetes.io/projected/f32356b3-4afc-4427-b684-e23c7a7838fb-kube-api-access-29nz7\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.424931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.416808    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32356b3-4afc-4427-b684-e23c7a7838fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.424931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.416882    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpk28\" (UniqueName: \"kubernetes.io/projected/e0b7b142-012c-4838-81a7-2404a362caa6-kube-api-access-hpk28\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.460360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.460322    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"]
Nov 03 01:00:00.470937 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.470906    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"]
Nov 03 01:00:00.471179 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.471052    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:00.483062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.483041    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"]
Nov 03 01:00:00.493397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.493369    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"]
Nov 03 01:00:00.493549 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.493529    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521172    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b7b142-012c-4838-81a7-2404a362caa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521239    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lntq4\" (UniqueName: \"kubernetes.io/projected/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-kube-api-access-lntq4\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521470    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521518    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29nz7\" (UniqueName: \"kubernetes.io/projected/f32356b3-4afc-4427-b684-e23c7a7838fb-kube-api-access-29nz7\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521587    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c3046-9656-45e9-ae52-9fc2019b6cd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521684    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32356b3-4afc-4427-b684-e23c7a7838fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521787    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpk28\" (UniqueName: \"kubernetes.io/projected/e0b7b142-012c-4838-81a7-2404a362caa6-kube-api-access-hpk28\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.522708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.521835    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9srtq\" (UniqueName: \"kubernetes.io/projected/031c3046-9656-45e9-ae52-9fc2019b6cd2-kube-api-access-9srtq\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:00.523340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.523235    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b7b142-012c-4838-81a7-2404a362caa6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.524048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.524026    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.524494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.524468    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"]
Nov 03 01:00:00.524603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.524520    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"]
Nov 03 01:00:00.524603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.524547    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"]
Nov 03 01:00:00.524721 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.524642    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:00.524810 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.524791    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32356b3-4afc-4427-b684-e23c7a7838fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.545465 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.545432    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"]
Nov 03 01:00:00.564620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.564585    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29nz7\" (UniqueName: \"kubernetes.io/projected/f32356b3-4afc-4427-b684-e23c7a7838fb-kube-api-access-29nz7\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.565483 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.565457    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lntq4\" (UniqueName: \"kubernetes.io/projected/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-kube-api-access-lntq4\") pod \"maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.571269 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.567739    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpk28\" (UniqueName: \"kubernetes.io/projected/e0b7b142-012c-4838-81a7-2404a362caa6-kube-api-access-hpk28\") pod \"maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.623619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.623576    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r42m\" (UniqueName: \"kubernetes.io/projected/b2e711e9-1280-4d26-b552-4a34bb3544b3-kube-api-access-9r42m\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:00.623871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.623843    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdzmk\" (UniqueName: \"kubernetes.io/projected/bd9ac712-8859-4943-8cc0-0056cdba1619-kube-api-access-mdzmk\") pod \"maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:00.623971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.623940    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e711e9-1280-4d26-b552-4a34bb3544b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:00.624064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.624037    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a1068c-7f4a-4fed-94c1-d23f46614ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.624161 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.624136    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9flw\" (UniqueName: \"kubernetes.io/projected/56a1068c-7f4a-4fed-94c1-d23f46614ea8-kube-api-access-d9flw\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.624220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.624192    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c3046-9656-45e9-ae52-9fc2019b6cd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:00.624320 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.624233    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ac712-8859-4943-8cc0-0056cdba1619-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:00.624397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.624359    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9srtq\" (UniqueName: \"kubernetes.io/projected/031c3046-9656-45e9-ae52-9fc2019b6cd2-kube-api-access-9srtq\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:00.624970 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.624945    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c3046-9656-45e9-ae52-9fc2019b6cd2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.712535    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.726193    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r42m\" (UniqueName: \"kubernetes.io/projected/b2e711e9-1280-4d26-b552-4a34bb3544b3-kube-api-access-9r42m\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.726444    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdzmk\" (UniqueName: \"kubernetes.io/projected/bd9ac712-8859-4943-8cc0-0056cdba1619-kube-api-access-mdzmk\") pod \"maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.726528    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e711e9-1280-4d26-b552-4a34bb3544b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.726566    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a1068c-7f4a-4fed-94c1-d23f46614ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.726625    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9flw\" (UniqueName: \"kubernetes.io/projected/56a1068c-7f4a-4fed-94c1-d23f46614ea8-kube-api-access-d9flw\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.726675    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ac712-8859-4943-8cc0-0056cdba1619-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.727613    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ac712-8859-4943-8cc0-0056cdba1619-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:00.724464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.728730    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e711e9-1280-4d26-b552-4a34bb3544b3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:00.746368 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.746338    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:00.746866 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.746849    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:00.786441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.786401    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a1068c-7f4a-4fed-94c1-d23f46614ea8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.798712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.790892    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdzmk\" (UniqueName: \"kubernetes.io/projected/bd9ac712-8859-4943-8cc0-0056cdba1619-kube-api-access-mdzmk\") pod \"maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:00.808580 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.800876    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9srtq\" (UniqueName: \"kubernetes.io/projected/031c3046-9656-45e9-ae52-9fc2019b6cd2-kube-api-access-9srtq\") pod \"maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:00.809009 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.808982    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9flw\" (UniqueName: \"kubernetes.io/projected/56a1068c-7f4a-4fed-94c1-d23f46614ea8-kube-api-access-d9flw\") pod \"maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.814086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.814001    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r42m\" (UniqueName: \"kubernetes.io/projected/b2e711e9-1280-4d26-b552-4a34bb3544b3-kube-api-access-9r42m\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:00.829495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.829450    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:00.849543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:00.849509    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:01.066778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.066744    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:01.093873 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.093840    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:01.376952 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:00:01.376923    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-9118ec78305697b201d8c0e357f8aa6f6a35a2ca08b44f56af37f39aea153295 WatchSource:0}: Error finding container 9118ec78305697b201d8c0e357f8aa6f6a35a2ca08b44f56af37f39aea153295: Status 404 returned error can't find the container with id 9118ec78305697b201d8c0e357f8aa6f6a35a2ca08b44f56af37f39aea153295
Nov 03 01:00:01.378946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.378907    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"]
Nov 03 01:00:01.385815 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:00:01.385786    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8b1f0ed_acf8_4c9a_94a0_ce2f18d2aa0e.slice/crio-6aec7b7405a27c5f66440ef66198591e012d32718cd0e51d88a5273b07dd75ee WatchSource:0}: Error finding container 6aec7b7405a27c5f66440ef66198591e012d32718cd0e51d88a5273b07dd75ee: Status 404 returned error can't find the container with id 6aec7b7405a27c5f66440ef66198591e012d32718cd0e51d88a5273b07dd75ee
Nov 03 01:00:01.387530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.387484    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"]
Nov 03 01:00:01.410449 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:00:01.410423    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod031c3046_9656_45e9_ae52_9fc2019b6cd2.slice/crio-6e764ff621517845454d25b0f7d8fdc42b78548efaa2e360a8ee325936361537 WatchSource:0}: Error finding container 6e764ff621517845454d25b0f7d8fdc42b78548efaa2e360a8ee325936361537: Status 404 returned error can't find the container with id 6e764ff621517845454d25b0f7d8fdc42b78548efaa2e360a8ee325936361537
Nov 03 01:00:01.412838 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.412809    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"]
Nov 03 01:00:01.434780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.433347    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"]
Nov 03 01:00:01.445410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.445383    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"]
Nov 03 01:00:01.451631 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.451578    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"]
Nov 03 01:00:01.456730 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:00:01.456704    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9ac712_8859_4943_8cc0_0056cdba1619.slice/crio-fa1a4ce0b2ec0daf6e70c260295b589f7622e9b61c7efec0b2cba510d114bd22 WatchSource:0}: Error finding container fa1a4ce0b2ec0daf6e70c260295b589f7622e9b61c7efec0b2cba510d114bd22: Status 404 returned error can't find the container with id fa1a4ce0b2ec0daf6e70c260295b589f7622e9b61c7efec0b2cba510d114bd22
Nov 03 01:00:01.464927 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:01.464898    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"]
Nov 03 01:00:01.468136 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:00:01.468109    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a1068c_7f4a_4fed_94c1_d23f46614ea8.slice/crio-dff319b8927a142575de0cb5937b197305f4e723f316b6edbfe948a656500917 WatchSource:0}: Error finding container dff319b8927a142575de0cb5937b197305f4e723f316b6edbfe948a656500917: Status 404 returned error can't find the container with id dff319b8927a142575de0cb5937b197305f4e723f316b6edbfe948a656500917
Nov 03 01:00:02.304656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.304622    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerStarted","Data":"848dd619ef31d7874ea6ca8bbe2c8073d012a835a36744084bb8fcc58a00492e"}
Nov 03 01:00:02.304797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.304722    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerStarted","Data":"6aec7b7405a27c5f66440ef66198591e012d32718cd0e51d88a5273b07dd75ee"}
Nov 03 01:00:02.306308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.306284    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerStarted","Data":"f02060b07ed600b7fa871902bac290e8cb7198847f934a44d4d4afa699aa663a"}
Nov 03 01:00:02.306397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.306331    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerStarted","Data":"6e764ff621517845454d25b0f7d8fdc42b78548efaa2e360a8ee325936361537"}
Nov 03 01:00:02.307873 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.307848    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" event={"ID":"bd9ac712-8859-4943-8cc0-0056cdba1619","Type":"ContainerStarted","Data":"fa1a4ce0b2ec0daf6e70c260295b589f7622e9b61c7efec0b2cba510d114bd22"}
Nov 03 01:00:02.309410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.309390    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerStarted","Data":"bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b"}
Nov 03 01:00:02.309502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.309431    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerStarted","Data":"9118ec78305697b201d8c0e357f8aa6f6a35a2ca08b44f56af37f39aea153295"}
Nov 03 01:00:02.311017 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.310997    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" event={"ID":"56a1068c-7f4a-4fed-94c1-d23f46614ea8","Type":"ContainerStarted","Data":"dff319b8927a142575de0cb5937b197305f4e723f316b6edbfe948a656500917"}
Nov 03 01:00:02.312600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.312582    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerStarted","Data":"eccb9397519659229a229ae7fa1539b4f73a5b787af039ce440ef8d91ee6d82a"}
Nov 03 01:00:02.312697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.312620    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerStarted","Data":"26815d3e16f6ba7999d7e3c70561d1d6cb871cb2f48d9ad40546fb94b678838e"}
Nov 03 01:00:02.313983 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.313959    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerStarted","Data":"df19c49af0e34059e3028ce4d78566d3fd99f032939a406f8e1bc1c228d0a4b2"}
Nov 03 01:00:02.314076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.313999    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerStarted","Data":"736dabb29fc9c7307ab2eff14f4c97ea8ee1c75696ee3603b2a87f768d412e12"}
Nov 03 01:00:02.341214 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.341159    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" podStartSLOduration=2.341143251 podStartE2EDuration="2.341143251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.339602643 +0000 UTC m=+3239.894954666" watchObservedRunningTime="2026-11-03 01:00:02.341143251 +0000 UTC m=+3239.896495278"
Nov 03 01:00:02.379419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.379368    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" podStartSLOduration=2.379353563 podStartE2EDuration="2.379353563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.375065642 +0000 UTC m=+3239.930417667" watchObservedRunningTime="2026-11-03 01:00:02.379353563 +0000 UTC m=+3239.934705587"
Nov 03 01:00:02.414290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.414217    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" podStartSLOduration=2.41420245 podStartE2EDuration="2.41420245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.412916374 +0000 UTC m=+3239.968268397" watchObservedRunningTime="2026-11-03 01:00:02.41420245 +0000 UTC m=+3239.969554465"
Nov 03 01:00:02.454019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.453962    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" podStartSLOduration=2.45394692 podStartE2EDuration="2.45394692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.448992199 +0000 UTC m=+3240.004344224" watchObservedRunningTime="2026-11-03 01:00:02.45394692 +0000 UTC m=+3240.009299002"
Nov 03 01:00:02.496961 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.496412    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" podStartSLOduration=2.496392568 podStartE2EDuration="2.496392568s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.490023683 +0000 UTC m=+3240.045375707" watchObservedRunningTime="2026-11-03 01:00:02.496392568 +0000 UTC m=+3240.051744593"
Nov 03 01:00:02.529372 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.529318    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" podStartSLOduration=2.529303462 podStartE2EDuration="2.529303462s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.525531515 +0000 UTC m=+3240.080883538" watchObservedRunningTime="2026-11-03 01:00:02.529303462 +0000 UTC m=+3240.084655486"
Nov 03 01:00:02.562352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:02.561719    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" podStartSLOduration=2.561703866 podStartE2EDuration="2.561703866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:02.561131028 +0000 UTC m=+3240.116483051" watchObservedRunningTime="2026-11-03 01:00:02.561703866 +0000 UTC m=+3240.117055890"
Nov 03 01:00:03.206205 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:03.206167    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:03.326603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.326568    6000 generic.go:356] "Generic (PLEG): container finished" podID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerID="f02060b07ed600b7fa871902bac290e8cb7198847f934a44d4d4afa699aa663a" exitCode=60
Nov 03 01:00:03.326827 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.326652    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerDied","Data":"f02060b07ed600b7fa871902bac290e8cb7198847f934a44d4d4afa699aa663a"}
Nov 03 01:00:03.327081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.327061    6000 scope.go:122] "RemoveContainer" containerID="f02060b07ed600b7fa871902bac290e8cb7198847f934a44d4d4afa699aa663a"
Nov 03 01:00:03.328593 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.328567    6000 generic.go:356] "Generic (PLEG): container finished" podID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerID="8a5d9e2794a4a69f4dc875ef63cd4a321f76cece76e48662f4b4bfeba023bc0c" exitCode=60
Nov 03 01:00:03.328733 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.328644    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" event={"ID":"bd9ac712-8859-4943-8cc0-0056cdba1619","Type":"ContainerDied","Data":"8a5d9e2794a4a69f4dc875ef63cd4a321f76cece76e48662f4b4bfeba023bc0c"}
Nov 03 01:00:03.328938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.328923    6000 scope.go:122] "RemoveContainer" containerID="8a5d9e2794a4a69f4dc875ef63cd4a321f76cece76e48662f4b4bfeba023bc0c"
Nov 03 01:00:03.330980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.330927    6000 generic.go:356] "Generic (PLEG): container finished" podID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerID="bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b" exitCode=60
Nov 03 01:00:03.331185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.330984    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerDied","Data":"bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b"}
Nov 03 01:00:03.331386 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.331362    6000 scope.go:122] "RemoveContainer" containerID="bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b"
Nov 03 01:00:03.333520 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.333496    6000 generic.go:356] "Generic (PLEG): container finished" podID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerID="32280536c50fc4f4018f5fc6b4324e03460c1695936d85b1a0e8eecd1772e410" exitCode=60
Nov 03 01:00:03.334057 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.333578    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" event={"ID":"56a1068c-7f4a-4fed-94c1-d23f46614ea8","Type":"ContainerDied","Data":"32280536c50fc4f4018f5fc6b4324e03460c1695936d85b1a0e8eecd1772e410"}
Nov 03 01:00:03.334057 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.333913    6000 scope.go:122] "RemoveContainer" containerID="32280536c50fc4f4018f5fc6b4324e03460c1695936d85b1a0e8eecd1772e410"
Nov 03 01:00:03.336191 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.336168    6000 generic.go:356] "Generic (PLEG): container finished" podID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerID="eccb9397519659229a229ae7fa1539b4f73a5b787af039ce440ef8d91ee6d82a" exitCode=60
Nov 03 01:00:03.336313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.336300    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerDied","Data":"eccb9397519659229a229ae7fa1539b4f73a5b787af039ce440ef8d91ee6d82a"}
Nov 03 01:00:03.336505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.336487    6000 scope.go:122] "RemoveContainer" containerID="eccb9397519659229a229ae7fa1539b4f73a5b787af039ce440ef8d91ee6d82a"
Nov 03 01:00:03.338451 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.338432    6000 generic.go:356] "Generic (PLEG): container finished" podID="e0b7b142-012c-4838-81a7-2404a362caa6" containerID="df19c49af0e34059e3028ce4d78566d3fd99f032939a406f8e1bc1c228d0a4b2" exitCode=60
Nov 03 01:00:03.338581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.338472    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerDied","Data":"df19c49af0e34059e3028ce4d78566d3fd99f032939a406f8e1bc1c228d0a4b2"}
Nov 03 01:00:03.338743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.338728    6000 scope.go:122] "RemoveContainer" containerID="df19c49af0e34059e3028ce4d78566d3fd99f032939a406f8e1bc1c228d0a4b2"
Nov 03 01:00:03.340232 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.340211    6000 generic.go:356] "Generic (PLEG): container finished" podID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerID="848dd619ef31d7874ea6ca8bbe2c8073d012a835a36744084bb8fcc58a00492e" exitCode=60
Nov 03 01:00:03.340379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.340271    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerDied","Data":"848dd619ef31d7874ea6ca8bbe2c8073d012a835a36744084bb8fcc58a00492e"}
Nov 03 01:00:03.340678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:03.340661    6000 scope.go:122] "RemoveContainer" containerID="848dd619ef31d7874ea6ca8bbe2c8073d012a835a36744084bb8fcc58a00492e"
Nov 03 01:00:04.359691 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:04.359656    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" event={"ID":"bd9ac712-8859-4943-8cc0-0056cdba1619","Type":"ContainerStarted","Data":"e986e12b9da9b1347e10bda8f2540c6a068b67e78af4f4b676973b4216cc32e2"}
Nov 03 01:00:04.362090 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:04.362063    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerStarted","Data":"f04b021011e7b734ff93c83334cb76ee540b7a2eb04592adb640ec17abae25d4"}
Nov 03 01:00:04.363949 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:04.363924    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerStarted","Data":"599771d6706dd158fa3e3b138d7fdfba1d7f1221525a57fc4a06f4cf1400ed86"}
Nov 03 01:00:04.366085 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:04.366059    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerStarted","Data":"ee4cc33a517c1c5d42b5761d421263488c4ca26569ae1a85b02dc0559420d9be"}
Nov 03 01:00:04.368560 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:04.368535    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerStarted","Data":"99d72380b95583c9850fe70dce65ce1f9939762cf82d7bfc761677405516cae9"}
Nov 03 01:00:04.370833 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:04.370814    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerStarted","Data":"fcc4a6224a41e11cc1013eeb011043099b14f631f5c42259da2af7c71d8aa392"}
Nov 03 01:00:05.382052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.382015    6000 generic.go:356] "Generic (PLEG): container finished" podID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerID="f04b021011e7b734ff93c83334cb76ee540b7a2eb04592adb640ec17abae25d4" exitCode=60
Nov 03 01:00:05.382585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.382065    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerDied","Data":"f04b021011e7b734ff93c83334cb76ee540b7a2eb04592adb640ec17abae25d4"}
Nov 03 01:00:05.382585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.382141    6000 scope.go:122] "RemoveContainer" containerID="bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b"
Nov 03 01:00:05.382585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.382512    6000 scope.go:122] "RemoveContainer" containerID="f04b021011e7b734ff93c83334cb76ee540b7a2eb04592adb640ec17abae25d4"
Nov 03 01:00:05.383030 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:05.383001    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn_odh-ai-gateway-infra(f32356b3-4afc-4427-b684-e23c7a7838fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb"
Nov 03 01:00:05.384473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.384450    6000 generic.go:356] "Generic (PLEG): container finished" podID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerID="1c609e1bb1774bc6c0c3a0bb19108760b9029083772a98811f945cf2817856bc" exitCode=60
Nov 03 01:00:05.384594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.384562    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" event={"ID":"56a1068c-7f4a-4fed-94c1-d23f46614ea8","Type":"ContainerDied","Data":"1c609e1bb1774bc6c0c3a0bb19108760b9029083772a98811f945cf2817856bc"}
Nov 03 01:00:05.385051 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.385029    6000 scope.go:122] "RemoveContainer" containerID="1c609e1bb1774bc6c0c3a0bb19108760b9029083772a98811f945cf2817856bc"
Nov 03 01:00:05.385548 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:05.385524    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z_odh-ai-gateway-infra(56a1068c-7f4a-4fed-94c1-d23f46614ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8"
Nov 03 01:00:05.387081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.387057    6000 generic.go:356] "Generic (PLEG): container finished" podID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerID="599771d6706dd158fa3e3b138d7fdfba1d7f1221525a57fc4a06f4cf1400ed86" exitCode=60
Nov 03 01:00:05.387210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.387097    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerDied","Data":"599771d6706dd158fa3e3b138d7fdfba1d7f1221525a57fc4a06f4cf1400ed86"}
Nov 03 01:00:05.387420 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.387405    6000 scope.go:122] "RemoveContainer" containerID="599771d6706dd158fa3e3b138d7fdfba1d7f1221525a57fc4a06f4cf1400ed86"
Nov 03 01:00:05.387746 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:05.387725    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc_odh-ai-gateway-infra(b2e711e9-1280-4d26-b552-4a34bb3544b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3"
Nov 03 01:00:05.389164 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.389144    6000 generic.go:356] "Generic (PLEG): container finished" podID="e0b7b142-012c-4838-81a7-2404a362caa6" containerID="ee4cc33a517c1c5d42b5761d421263488c4ca26569ae1a85b02dc0559420d9be" exitCode=60
Nov 03 01:00:05.389306 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.389227    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerDied","Data":"ee4cc33a517c1c5d42b5761d421263488c4ca26569ae1a85b02dc0559420d9be"}
Nov 03 01:00:05.389488 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.389470    6000 scope.go:122] "RemoveContainer" containerID="ee4cc33a517c1c5d42b5761d421263488c4ca26569ae1a85b02dc0559420d9be"
Nov 03 01:00:05.389779 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:05.389758    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh_odh-ai-gateway-infra(e0b7b142-012c-4838-81a7-2404a362caa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" podUID="e0b7b142-012c-4838-81a7-2404a362caa6"
Nov 03 01:00:05.391466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.391447    6000 generic.go:356] "Generic (PLEG): container finished" podID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerID="99d72380b95583c9850fe70dce65ce1f9939762cf82d7bfc761677405516cae9" exitCode=60
Nov 03 01:00:05.391617 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.391520    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerDied","Data":"99d72380b95583c9850fe70dce65ce1f9939762cf82d7bfc761677405516cae9"}
Nov 03 01:00:05.391741 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.391727    6000 scope.go:122] "RemoveContainer" containerID="99d72380b95583c9850fe70dce65ce1f9939762cf82d7bfc761677405516cae9"
Nov 03 01:00:05.392039 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:05.392010    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd_odh-ai-gateway-infra(b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e"
Nov 03 01:00:05.393553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.393531    6000 generic.go:356] "Generic (PLEG): container finished" podID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerID="fcc4a6224a41e11cc1013eeb011043099b14f631f5c42259da2af7c71d8aa392" exitCode=60
Nov 03 01:00:05.393655 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.393609    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerDied","Data":"fcc4a6224a41e11cc1013eeb011043099b14f631f5c42259da2af7c71d8aa392"}
Nov 03 01:00:05.393817 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.393801    6000 scope.go:122] "RemoveContainer" containerID="fcc4a6224a41e11cc1013eeb011043099b14f631f5c42259da2af7c71d8aa392"
Nov 03 01:00:05.394123 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:05.394097    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq_odh-ai-gateway-infra(031c3046-9656-45e9-ae52-9fc2019b6cd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2"
Nov 03 01:00:05.395913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.395893    6000 generic.go:356] "Generic (PLEG): container finished" podID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerID="e986e12b9da9b1347e10bda8f2540c6a068b67e78af4f4b676973b4216cc32e2" exitCode=60
Nov 03 01:00:05.396020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.395921    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" event={"ID":"bd9ac712-8859-4943-8cc0-0056cdba1619","Type":"ContainerDied","Data":"e986e12b9da9b1347e10bda8f2540c6a068b67e78af4f4b676973b4216cc32e2"}
Nov 03 01:00:05.396310 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.396289    6000 scope.go:122] "RemoveContainer" containerID="e986e12b9da9b1347e10bda8f2540c6a068b67e78af4f4b676973b4216cc32e2"
Nov 03 01:00:05.396638 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:05.396617    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n_odh-ai-gateway-infra(bd9ac712-8859-4943-8cc0-0056cdba1619)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619"
Nov 03 01:00:05.405136 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.405114    6000 scope.go:122] "RemoveContainer" containerID="32280536c50fc4f4018f5fc6b4324e03460c1695936d85b1a0e8eecd1772e410"
Nov 03 01:00:05.419408 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.419389    6000 scope.go:122] "RemoveContainer" containerID="eccb9397519659229a229ae7fa1539b4f73a5b787af039ce440ef8d91ee6d82a"
Nov 03 01:00:05.433466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.433441    6000 scope.go:122] "RemoveContainer" containerID="df19c49af0e34059e3028ce4d78566d3fd99f032939a406f8e1bc1c228d0a4b2"
Nov 03 01:00:05.448155 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.448129    6000 scope.go:122] "RemoveContainer" containerID="848dd619ef31d7874ea6ca8bbe2c8073d012a835a36744084bb8fcc58a00492e"
Nov 03 01:00:05.464821 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.464796    6000 scope.go:122] "RemoveContainer" containerID="f02060b07ed600b7fa871902bac290e8cb7198847f934a44d4d4afa699aa663a"
Nov 03 01:00:05.481077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:05.481054    6000 scope.go:122] "RemoveContainer" containerID="8a5d9e2794a4a69f4dc875ef63cd4a321f76cece76e48662f4b4bfeba023bc0c"
Nov 03 01:00:06.406975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:06.406944    6000 scope.go:122] "RemoveContainer" containerID="f04b021011e7b734ff93c83334cb76ee540b7a2eb04592adb640ec17abae25d4"
Nov 03 01:00:06.407466 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:06.407415    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn_odh-ai-gateway-infra(f32356b3-4afc-4427-b684-e23c7a7838fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb"
Nov 03 01:00:06.408606 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:06.408586    6000 scope.go:122] "RemoveContainer" containerID="1c609e1bb1774bc6c0c3a0bb19108760b9029083772a98811f945cf2817856bc"
Nov 03 01:00:06.408933 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:06.408910    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z_odh-ai-gateway-infra(56a1068c-7f4a-4fed-94c1-d23f46614ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8"
Nov 03 01:00:06.410139 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:06.410105    6000 scope.go:122] "RemoveContainer" containerID="599771d6706dd158fa3e3b138d7fdfba1d7f1221525a57fc4a06f4cf1400ed86"
Nov 03 01:00:06.410440 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:06.410421    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc_odh-ai-gateway-infra(b2e711e9-1280-4d26-b552-4a34bb3544b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3"
Nov 03 01:00:06.411719 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:06.411700    6000 scope.go:122] "RemoveContainer" containerID="ee4cc33a517c1c5d42b5761d421263488c4ca26569ae1a85b02dc0559420d9be"
Nov 03 01:00:06.412004 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:06.411984    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh_odh-ai-gateway-infra(e0b7b142-012c-4838-81a7-2404a362caa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" podUID="e0b7b142-012c-4838-81a7-2404a362caa6"
Nov 03 01:00:06.413268 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:06.413238    6000 scope.go:122] "RemoveContainer" containerID="99d72380b95583c9850fe70dce65ce1f9939762cf82d7bfc761677405516cae9"
Nov 03 01:00:06.413499 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:06.413481    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd_odh-ai-gateway-infra(b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e"
Nov 03 01:00:06.414874 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:06.414852    6000 scope.go:122] "RemoveContainer" containerID="fcc4a6224a41e11cc1013eeb011043099b14f631f5c42259da2af7c71d8aa392"
Nov 03 01:00:06.415097 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:06.415079    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq_odh-ai-gateway-infra(031c3046-9656-45e9-ae52-9fc2019b6cd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2"
Nov 03 01:00:06.416371 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:06.416355    6000 scope.go:122] "RemoveContainer" containerID="e986e12b9da9b1347e10bda8f2540c6a068b67e78af4f4b676973b4216cc32e2"
Nov 03 01:00:06.416579 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:06.416560    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n_odh-ai-gateway-infra(bd9ac712-8859-4943-8cc0-0056cdba1619)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619"
Nov 03 01:00:10.309196 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:10.309162    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:15.559720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:15.559642    6000 scope.go:122] "RemoveContainer" containerID="ee4cc33a517c1c5d42b5761d421263488c4ca26569ae1a85b02dc0559420d9be"
Nov 03 01:00:15.560093 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:15.559745    6000 scope.go:122] "RemoveContainer" containerID="599771d6706dd158fa3e3b138d7fdfba1d7f1221525a57fc4a06f4cf1400ed86"
Nov 03 01:00:15.593581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:15.593551    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k" podUID="9163c109-b64a-4e78-943f-c8e73a20682b" containerName="main" containerID="cri-o://8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3" gracePeriod=45
Nov 03 01:00:15.929582 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:15.929557    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 01:00:16.050648 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.050615    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-home\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-home\") pod \"9163c109-b64a-4e78-943f-c8e73a20682b\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") "
Nov 03 01:00:16.050793 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.050724    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-dshm\") pod \"9163c109-b64a-4e78-943f-c8e73a20682b\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") "
Nov 03 01:00:16.050868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.050850    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9163c109-b64a-4e78-943f-c8e73a20682b-kube-api-access-pwcrw\" (UniqueName: \"kubernetes.io/projected/9163c109-b64a-4e78-943f-c8e73a20682b-kube-api-access-pwcrw\") pod \"9163c109-b64a-4e78-943f-c8e73a20682b\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") "
Nov 03 01:00:16.050940 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.050870    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-home" pod "9163c109-b64a-4e78-943f-c8e73a20682b" (UID: "9163c109-b64a-4e78-943f-c8e73a20682b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:16.051034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.051016    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9163c109-b64a-4e78-943f-c8e73a20682b-tls-certs\" (UniqueName: \"kubernetes.io/secret/9163c109-b64a-4e78-943f-c8e73a20682b-tls-certs\") pod \"9163c109-b64a-4e78-943f-c8e73a20682b\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") "
Nov 03 01:00:16.051190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.051176    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-tmp-dir\") pod \"9163c109-b64a-4e78-943f-c8e73a20682b\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") "
Nov 03 01:00:16.051360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.051335    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-model-cache\") pod \"9163c109-b64a-4e78-943f-c8e73a20682b\" (UID: \"9163c109-b64a-4e78-943f-c8e73a20682b\") "
Nov 03 01:00:16.051748 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.051635    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-tmp-dir" pod "9163c109-b64a-4e78-943f-c8e73a20682b" (UID: "9163c109-b64a-4e78-943f-c8e73a20682b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:16.051860 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.051660    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-model-cache" pod "9163c109-b64a-4e78-943f-c8e73a20682b" (UID: "9163c109-b64a-4e78-943f-c8e73a20682b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:16.052417 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.052393    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:16.052521 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.052421    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:16.052521 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.052437    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:16.053188 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.053163    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-dshm" pod "9163c109-b64a-4e78-943f-c8e73a20682b" (UID: "9163c109-b64a-4e78-943f-c8e73a20682b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:00:16.053314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.053300    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9163c109-b64a-4e78-943f-c8e73a20682b-tls-certs" pod "9163c109-b64a-4e78-943f-c8e73a20682b" (UID: "9163c109-b64a-4e78-943f-c8e73a20682b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:00:16.053394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.053374    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9163c109-b64a-4e78-943f-c8e73a20682b-kube-api-access-pwcrw" pod "9163c109-b64a-4e78-943f-c8e73a20682b" (UID: "9163c109-b64a-4e78-943f-c8e73a20682b"). InnerVolumeSpecName "kube-api-access-pwcrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:16.154997 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.154643    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9163c109-b64a-4e78-943f-c8e73a20682b-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:16.154997 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.154680    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwcrw\" (UniqueName: \"kubernetes.io/projected/9163c109-b64a-4e78-943f-c8e73a20682b-kube-api-access-pwcrw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:16.154997 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.154707    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9163c109-b64a-4e78-943f-c8e73a20682b-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:16.560124 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.560040    6000 scope.go:122] "RemoveContainer" containerID="e986e12b9da9b1347e10bda8f2540c6a068b67e78af4f4b676973b4216cc32e2"
Nov 03 01:00:16.560639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.560288    6000 scope.go:122] "RemoveContainer" containerID="fcc4a6224a41e11cc1013eeb011043099b14f631f5c42259da2af7c71d8aa392"
Nov 03 01:00:16.560639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.560408    6000 scope.go:122] "RemoveContainer" containerID="f04b021011e7b734ff93c83334cb76ee540b7a2eb04592adb640ec17abae25d4"
Nov 03 01:00:16.582336 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.582302    6000 generic.go:356] "Generic (PLEG): container finished" podID="9163c109-b64a-4e78-943f-c8e73a20682b" containerID="8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3" exitCode=0
Nov 03 01:00:16.582497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.582376    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"
Nov 03 01:00:16.582497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.582461    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k" event={"ID":"9163c109-b64a-4e78-943f-c8e73a20682b","Type":"ContainerDied","Data":"8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3"}
Nov 03 01:00:16.582619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.582497    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k" event={"ID":"9163c109-b64a-4e78-943f-c8e73a20682b","Type":"ContainerDied","Data":"4e5b74a6c9693c19e8bb1a937063cbabea4dfa0ba12b5c9b750bde49382b4739"}
Nov 03 01:00:16.582619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.582513    6000 scope.go:122] "RemoveContainer" containerID="8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3"
Nov 03 01:00:16.585021 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.584996    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerStarted","Data":"798882084be823b665d396cf6026ddfe723eae07bc2326696c0f9b8ef9dbf209"}
Nov 03 01:00:16.590081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.590038    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerStarted","Data":"3571b0bba98d124cce4e6655c1abdd401dac2ba7dfc221977db9ceed44707e42"}
Nov 03 01:00:16.640815 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.640789    6000 scope.go:122] "RemoveContainer" containerID="8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3"
Nov 03 01:00:16.641207 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:16.641184    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3\": container with ID starting with 8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3 not found: ID does not exist" containerID="8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3"
Nov 03 01:00:16.641295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.641216    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3"} err="failed to get container status \"8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3\": rpc error: code = NotFound desc = could not find container \"8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3\": container with ID starting with 8f129995f0fbb0637568237363c000d1bf7e0fa34485814e3547c0f3539a3ed3 not found: ID does not exist"
Nov 03 01:00:16.706248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.706216    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"]
Nov 03 01:00:16.710474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.710455    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k"]
Nov 03 01:00:16.907172 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:16.907142    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-84f605/test-model-84f605-kserve-6b8d595889-vcj6k" podUID="9163c109-b64a-4e78-943f-c8e73a20682b" containerName="main" probeResult="failure" output="Get \"https://10.134.1.3:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:00:17.608404 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.608363    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerStarted","Data":"0ffedbc5b95959be7e38d47a07d45a825c55d4c54906e5fa91f48368013c089f"}
Nov 03 01:00:17.608924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.608902    6000 scope.go:122] "RemoveContainer" containerID="0ffedbc5b95959be7e38d47a07d45a825c55d4c54906e5fa91f48368013c089f"
Nov 03 01:00:17.609435 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:17.609407    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn_odh-ai-gateway-infra(f32356b3-4afc-4427-b684-e23c7a7838fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb"
Nov 03 01:00:17.610768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.610745    6000 generic.go:356] "Generic (PLEG): container finished" podID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerID="798882084be823b665d396cf6026ddfe723eae07bc2326696c0f9b8ef9dbf209" exitCode=60
Nov 03 01:00:17.610903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.610814    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerDied","Data":"798882084be823b665d396cf6026ddfe723eae07bc2326696c0f9b8ef9dbf209"}
Nov 03 01:00:17.610903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.610875    6000 scope.go:122] "RemoveContainer" containerID="599771d6706dd158fa3e3b138d7fdfba1d7f1221525a57fc4a06f4cf1400ed86"
Nov 03 01:00:17.611366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.611344    6000 scope.go:122] "RemoveContainer" containerID="798882084be823b665d396cf6026ddfe723eae07bc2326696c0f9b8ef9dbf209"
Nov 03 01:00:17.611814 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:17.611786    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc_odh-ai-gateway-infra(b2e711e9-1280-4d26-b552-4a34bb3544b3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3"
Nov 03 01:00:17.613064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.613040    6000 generic.go:356] "Generic (PLEG): container finished" podID="e0b7b142-012c-4838-81a7-2404a362caa6" containerID="3571b0bba98d124cce4e6655c1abdd401dac2ba7dfc221977db9ceed44707e42" exitCode=60
Nov 03 01:00:17.613151 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.613071    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerDied","Data":"3571b0bba98d124cce4e6655c1abdd401dac2ba7dfc221977db9ceed44707e42"}
Nov 03 01:00:17.613500 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.613482    6000 scope.go:122] "RemoveContainer" containerID="3571b0bba98d124cce4e6655c1abdd401dac2ba7dfc221977db9ceed44707e42"
Nov 03 01:00:17.613912 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:17.613843    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh_odh-ai-gateway-infra(e0b7b142-012c-4838-81a7-2404a362caa6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" podUID="e0b7b142-012c-4838-81a7-2404a362caa6"
Nov 03 01:00:17.615722 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.615701    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerStarted","Data":"799d8d34f7a88f729105cd3a3079adf0a6a62674700ae8d0d6dc9070b987a321"}
Nov 03 01:00:17.619160 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.619139    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" event={"ID":"bd9ac712-8859-4943-8cc0-0056cdba1619","Type":"ContainerStarted","Data":"215f4958206414f6f4f5d5da70b8d087620d270cea73d71ba677f1f0a06c80b3"}
Nov 03 01:00:17.647018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.646987    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"]
Nov 03 01:00:17.669969 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.669949    6000 scope.go:122] "RemoveContainer" containerID="ee4cc33a517c1c5d42b5761d421263488c4ca26569ae1a85b02dc0559420d9be"
Nov 03 01:00:17.686071 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:17.686040    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"]
Nov 03 01:00:18.298049 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:18.297953    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:18.560004 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.559919    6000 scope.go:122] "RemoveContainer" containerID="99d72380b95583c9850fe70dce65ce1f9939762cf82d7bfc761677405516cae9"
Nov 03 01:00:18.572594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.572559    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9163c109-b64a-4e78-943f-c8e73a20682b" path="/var/lib/kubelet/pods/9163c109-b64a-4e78-943f-c8e73a20682b/volumes"
Nov 03 01:00:18.631040 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.631011    6000 generic.go:356] "Generic (PLEG): container finished" podID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerID="799d8d34f7a88f729105cd3a3079adf0a6a62674700ae8d0d6dc9070b987a321" exitCode=60
Nov 03 01:00:18.631584 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.631078    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerDied","Data":"799d8d34f7a88f729105cd3a3079adf0a6a62674700ae8d0d6dc9070b987a321"}
Nov 03 01:00:18.631584 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.631133    6000 scope.go:122] "RemoveContainer" containerID="fcc4a6224a41e11cc1013eeb011043099b14f631f5c42259da2af7c71d8aa392"
Nov 03 01:00:18.631584 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.631549    6000 scope.go:122] "RemoveContainer" containerID="799d8d34f7a88f729105cd3a3079adf0a6a62674700ae8d0d6dc9070b987a321"
Nov 03 01:00:18.631915 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:18.631890    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq_odh-ai-gateway-infra(031c3046-9656-45e9-ae52-9fc2019b6cd2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2"
Nov 03 01:00:18.633571 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.633547    6000 generic.go:356] "Generic (PLEG): container finished" podID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerID="215f4958206414f6f4f5d5da70b8d087620d270cea73d71ba677f1f0a06c80b3" exitCode=60
Nov 03 01:00:18.633680 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.633582    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" event={"ID":"bd9ac712-8859-4943-8cc0-0056cdba1619","Type":"ContainerDied","Data":"215f4958206414f6f4f5d5da70b8d087620d270cea73d71ba677f1f0a06c80b3"}
Nov 03 01:00:18.634020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.634002    6000 scope.go:122] "RemoveContainer" containerID="215f4958206414f6f4f5d5da70b8d087620d270cea73d71ba677f1f0a06c80b3"
Nov 03 01:00:18.634427 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:18.634380    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n_odh-ai-gateway-infra(bd9ac712-8859-4943-8cc0-0056cdba1619)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619"
Nov 03 01:00:18.635863 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.635839    6000 generic.go:356] "Generic (PLEG): container finished" podID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerID="0ffedbc5b95959be7e38d47a07d45a825c55d4c54906e5fa91f48368013c089f" exitCode=60
Nov 03 01:00:18.636134 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.635913    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerDied","Data":"0ffedbc5b95959be7e38d47a07d45a825c55d4c54906e5fa91f48368013c089f"}
Nov 03 01:00:18.636410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.636390    6000 scope.go:122] "RemoveContainer" containerID="0ffedbc5b95959be7e38d47a07d45a825c55d4c54906e5fa91f48368013c089f"
Nov 03 01:00:18.636676 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:18.636656    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn_odh-ai-gateway-infra(f32356b3-4afc-4427-b684-e23c7a7838fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb"
Nov 03 01:00:18.648416 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.648394    6000 scope.go:122] "RemoveContainer" containerID="e986e12b9da9b1347e10bda8f2540c6a068b67e78af4f4b676973b4216cc32e2"
Nov 03 01:00:18.662840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.662802    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"]
Nov 03 01:00:18.666492 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.666125    6000 scope.go:122] "RemoveContainer" containerID="f04b021011e7b734ff93c83334cb76ee540b7a2eb04592adb640ec17abae25d4"
Nov 03 01:00:18.709711 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.709672    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"]
Nov 03 01:00:18.823948 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.823870    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"]
Nov 03 01:00:18.875205 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.875178    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:18.950841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:18.950811    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:19.006792 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.006763    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2e711e9-1280-4d26-b552-4a34bb3544b3-kube-api-access-9r42m\" (UniqueName: \"kubernetes.io/projected/b2e711e9-1280-4d26-b552-4a34bb3544b3-kube-api-access-9r42m\") pod \"b2e711e9-1280-4d26-b552-4a34bb3544b3\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") "
Nov 03 01:00:19.006792 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.006800    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2e711e9-1280-4d26-b552-4a34bb3544b3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e711e9-1280-4d26-b552-4a34bb3544b3-maas-api-service-ca\") pod \"b2e711e9-1280-4d26-b552-4a34bb3544b3\" (UID: \"b2e711e9-1280-4d26-b552-4a34bb3544b3\") "
Nov 03 01:00:19.007843 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.007550    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2e711e9-1280-4d26-b552-4a34bb3544b3-maas-api-service-ca" pod "b2e711e9-1280-4d26-b552-4a34bb3544b3" (UID: "b2e711e9-1280-4d26-b552-4a34bb3544b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:19.009567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.009543    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2e711e9-1280-4d26-b552-4a34bb3544b3-kube-api-access-9r42m" pod "b2e711e9-1280-4d26-b552-4a34bb3544b3" (UID: "b2e711e9-1280-4d26-b552-4a34bb3544b3"). InnerVolumeSpecName "kube-api-access-9r42m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:19.108813 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.108789    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b7b142-012c-4838-81a7-2404a362caa6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b7b142-012c-4838-81a7-2404a362caa6-maas-api-service-ca\") pod \"e0b7b142-012c-4838-81a7-2404a362caa6\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") "
Nov 03 01:00:19.108975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.108958    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b7b142-012c-4838-81a7-2404a362caa6-kube-api-access-hpk28\" (UniqueName: \"kubernetes.io/projected/e0b7b142-012c-4838-81a7-2404a362caa6-kube-api-access-hpk28\") pod \"e0b7b142-012c-4838-81a7-2404a362caa6\" (UID: \"e0b7b142-012c-4838-81a7-2404a362caa6\") "
Nov 03 01:00:19.109077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.109056    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b7b142-012c-4838-81a7-2404a362caa6-maas-api-service-ca" pod "e0b7b142-012c-4838-81a7-2404a362caa6" (UID: "e0b7b142-012c-4838-81a7-2404a362caa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:19.109667 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.109645    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r42m\" (UniqueName: \"kubernetes.io/projected/b2e711e9-1280-4d26-b552-4a34bb3544b3-kube-api-access-9r42m\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:19.109746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.109670    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2e711e9-1280-4d26-b552-4a34bb3544b3-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:19.109746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.109679    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0b7b142-012c-4838-81a7-2404a362caa6-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:19.111097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.111077    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b7b142-012c-4838-81a7-2404a362caa6-kube-api-access-hpk28" pod "e0b7b142-012c-4838-81a7-2404a362caa6" (UID: "e0b7b142-012c-4838-81a7-2404a362caa6"). InnerVolumeSpecName "kube-api-access-hpk28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:19.211570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.211529    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpk28\" (UniqueName: \"kubernetes.io/projected/e0b7b142-012c-4838-81a7-2404a362caa6-kube-api-access-hpk28\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:19.560047 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.560016    6000 scope.go:122] "RemoveContainer" containerID="1c609e1bb1774bc6c0c3a0bb19108760b9029083772a98811f945cf2817856bc"
Nov 03 01:00:19.651675 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.651634    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerStarted","Data":"c21139eabec0e85e6a09fb3b827b9f62efda71424549cfdc9befd06b40afcdd1"}
Nov 03 01:00:19.659472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.659443    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc" event={"ID":"b2e711e9-1280-4d26-b552-4a34bb3544b3","Type":"ContainerDied","Data":"26815d3e16f6ba7999d7e3c70561d1d6cb871cb2f48d9ad40546fb94b678838e"}
Nov 03 01:00:19.659650 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.659636    6000 scope.go:122] "RemoveContainer" containerID="798882084be823b665d396cf6026ddfe723eae07bc2326696c0f9b8ef9dbf209"
Nov 03 01:00:19.659770 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.659460    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"
Nov 03 01:00:19.662141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.662114    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh" event={"ID":"e0b7b142-012c-4838-81a7-2404a362caa6","Type":"ContainerDied","Data":"736dabb29fc9c7307ab2eff14f4c97ea8ee1c75696ee3603b2a87f768d412e12"}
Nov 03 01:00:19.662230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.662206    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"
Nov 03 01:00:19.679859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.679774    6000 scope.go:122] "RemoveContainer" containerID="3571b0bba98d124cce4e6655c1abdd401dac2ba7dfc221977db9ceed44707e42"
Nov 03 01:00:19.901341 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.899413    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"]
Nov 03 01:00:19.908678 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:19.908040    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:00:19.910813 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.910713    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-d87781-29894460-2g2xh"]
Nov 03 01:00:19.940874 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.940808    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"]
Nov 03 01:00:19.946581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.946556    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-7ef1b6-29894460-mvfdc"]
Nov 03 01:00:19.967388 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:19.967361    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:20.061845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.061815    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/031c3046-9656-45e9-ae52-9fc2019b6cd2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c3046-9656-45e9-ae52-9fc2019b6cd2-maas-api-service-ca\") pod \"031c3046-9656-45e9-ae52-9fc2019b6cd2\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") "
Nov 03 01:00:20.062023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.061883    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/031c3046-9656-45e9-ae52-9fc2019b6cd2-kube-api-access-9srtq\" (UniqueName: \"kubernetes.io/projected/031c3046-9656-45e9-ae52-9fc2019b6cd2-kube-api-access-9srtq\") pod \"031c3046-9656-45e9-ae52-9fc2019b6cd2\" (UID: \"031c3046-9656-45e9-ae52-9fc2019b6cd2\") "
Nov 03 01:00:20.062312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.062239    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/031c3046-9656-45e9-ae52-9fc2019b6cd2-maas-api-service-ca" pod "031c3046-9656-45e9-ae52-9fc2019b6cd2" (UID: "031c3046-9656-45e9-ae52-9fc2019b6cd2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:20.062508 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.062487    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/031c3046-9656-45e9-ae52-9fc2019b6cd2-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.065279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.064637    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/031c3046-9656-45e9-ae52-9fc2019b6cd2-kube-api-access-9srtq" pod "031c3046-9656-45e9-ae52-9fc2019b6cd2" (UID: "031c3046-9656-45e9-ae52-9fc2019b6cd2"). InnerVolumeSpecName "kube-api-access-9srtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:20.118314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.118290    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:20.120645 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.120625    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:20.164696 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.164656    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd9ac712-8859-4943-8cc0-0056cdba1619-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ac712-8859-4943-8cc0-0056cdba1619-maas-api-service-ca\") pod \"bd9ac712-8859-4943-8cc0-0056cdba1619\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") "
Nov 03 01:00:20.164908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.164761    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd9ac712-8859-4943-8cc0-0056cdba1619-kube-api-access-mdzmk\" (UniqueName: \"kubernetes.io/projected/bd9ac712-8859-4943-8cc0-0056cdba1619-kube-api-access-mdzmk\") pod \"bd9ac712-8859-4943-8cc0-0056cdba1619\" (UID: \"bd9ac712-8859-4943-8cc0-0056cdba1619\") "
Nov 03 01:00:20.165075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.165050    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f32356b3-4afc-4427-b684-e23c7a7838fb-kube-api-access-29nz7\" (UniqueName: \"kubernetes.io/projected/f32356b3-4afc-4427-b684-e23c7a7838fb-kube-api-access-29nz7\") pod \"f32356b3-4afc-4427-b684-e23c7a7838fb\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") "
Nov 03 01:00:20.165170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.165125    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd9ac712-8859-4943-8cc0-0056cdba1619-maas-api-service-ca" pod "bd9ac712-8859-4943-8cc0-0056cdba1619" (UID: "bd9ac712-8859-4943-8cc0-0056cdba1619"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:20.165291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.165269    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f32356b3-4afc-4427-b684-e23c7a7838fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32356b3-4afc-4427-b684-e23c7a7838fb-maas-api-service-ca\") pod \"f32356b3-4afc-4427-b684-e23c7a7838fb\" (UID: \"f32356b3-4afc-4427-b684-e23c7a7838fb\") "
Nov 03 01:00:20.165815 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.165790    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f32356b3-4afc-4427-b684-e23c7a7838fb-maas-api-service-ca" pod "f32356b3-4afc-4427-b684-e23c7a7838fb" (UID: "f32356b3-4afc-4427-b684-e23c7a7838fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:20.166391 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.166371    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f32356b3-4afc-4427-b684-e23c7a7838fb-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.166482 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.166393    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd9ac712-8859-4943-8cc0-0056cdba1619-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.166482 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.166409    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9srtq\" (UniqueName: \"kubernetes.io/projected/031c3046-9656-45e9-ae52-9fc2019b6cd2-kube-api-access-9srtq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.167997 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.167976    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd9ac712-8859-4943-8cc0-0056cdba1619-kube-api-access-mdzmk" pod "bd9ac712-8859-4943-8cc0-0056cdba1619" (UID: "bd9ac712-8859-4943-8cc0-0056cdba1619"). InnerVolumeSpecName "kube-api-access-mdzmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:20.168062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.168049    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f32356b3-4afc-4427-b684-e23c7a7838fb-kube-api-access-29nz7" pod "f32356b3-4afc-4427-b684-e23c7a7838fb" (UID: "f32356b3-4afc-4427-b684-e23c7a7838fb"). InnerVolumeSpecName "kube-api-access-29nz7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:20.267873 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.267836    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29nz7\" (UniqueName: \"kubernetes.io/projected/f32356b3-4afc-4427-b684-e23c7a7838fb-kube-api-access-29nz7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.267873 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.267867    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdzmk\" (UniqueName: \"kubernetes.io/projected/bd9ac712-8859-4943-8cc0-0056cdba1619-kube-api-access-mdzmk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:20.329159 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:20.329122    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:20.571513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.571436    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" path="/var/lib/kubelet/pods/b2e711e9-1280-4d26-b552-4a34bb3544b3/volumes"
Nov 03 01:00:20.572007 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.571986    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" path="/var/lib/kubelet/pods/e0b7b142-012c-4838-81a7-2404a362caa6/volumes"
Nov 03 01:00:20.672089 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.672053    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" event={"ID":"56a1068c-7f4a-4fed-94c1-d23f46614ea8","Type":"ContainerStarted","Data":"1f1dc0f3a9f203dc42ddca28dd59fbe407f3ffdbd88bd8abf7ef2ce96e6f4c22"}
Nov 03 01:00:20.676297 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.676239    6000 generic.go:356] "Generic (PLEG): container finished" podID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerID="c21139eabec0e85e6a09fb3b827b9f62efda71424549cfdc9befd06b40afcdd1" exitCode=60
Nov 03 01:00:20.676297 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.676285    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerDied","Data":"c21139eabec0e85e6a09fb3b827b9f62efda71424549cfdc9befd06b40afcdd1"}
Nov 03 01:00:20.676475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.676341    6000 scope.go:122] "RemoveContainer" containerID="99d72380b95583c9850fe70dce65ce1f9939762cf82d7bfc761677405516cae9"
Nov 03 01:00:20.676707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.676692    6000 scope.go:122] "RemoveContainer" containerID="c21139eabec0e85e6a09fb3b827b9f62efda71424549cfdc9befd06b40afcdd1"
Nov 03 01:00:20.676987 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:20.676966    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd_odh-ai-gateway-infra(b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e"
Nov 03 01:00:20.677918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.677905    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"
Nov 03 01:00:20.678008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.677917    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq" event={"ID":"031c3046-9656-45e9-ae52-9fc2019b6cd2","Type":"ContainerDied","Data":"6e764ff621517845454d25b0f7d8fdc42b78548efaa2e360a8ee325936361537"}
Nov 03 01:00:20.679394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.679376    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"
Nov 03 01:00:20.679511 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.679417    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n" event={"ID":"bd9ac712-8859-4943-8cc0-0056cdba1619","Type":"ContainerDied","Data":"fa1a4ce0b2ec0daf6e70c260295b589f7622e9b61c7efec0b2cba510d114bd22"}
Nov 03 01:00:20.681112 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.681056    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"
Nov 03 01:00:20.681112 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.681091    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn" event={"ID":"f32356b3-4afc-4427-b684-e23c7a7838fb","Type":"ContainerDied","Data":"9118ec78305697b201d8c0e357f8aa6f6a35a2ca08b44f56af37f39aea153295"}
Nov 03 01:00:20.692219 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.692194    6000 scope.go:122] "RemoveContainer" containerID="799d8d34f7a88f729105cd3a3079adf0a6a62674700ae8d0d6dc9070b987a321"
Nov 03 01:00:20.705363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.705336    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"]
Nov 03 01:00:20.744829 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.744801    6000 scope.go:122] "RemoveContainer" containerID="215f4958206414f6f4f5d5da70b8d087620d270cea73d71ba677f1f0a06c80b3"
Nov 03 01:00:20.758384 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.758363    6000 scope.go:122] "RemoveContainer" containerID="0ffedbc5b95959be7e38d47a07d45a825c55d4c54906e5fa91f48368013c089f"
Nov 03 01:00:20.786401 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.786375    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"]
Nov 03 01:00:20.791494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.791472    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-2858fb-29894460-dtmzn"]
Nov 03 01:00:20.829268 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.829188    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"]
Nov 03 01:00:20.835659 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.835637    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-7f20f7-29894460-skh5n"]
Nov 03 01:00:20.867394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.867363    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"]
Nov 03 01:00:20.871920 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:20.871892    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-b85343-29894460-fllcq"]
Nov 03 01:00:21.693746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.693710    6000 generic.go:356] "Generic (PLEG): container finished" podID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerID="1f1dc0f3a9f203dc42ddca28dd59fbe407f3ffdbd88bd8abf7ef2ce96e6f4c22" exitCode=60
Nov 03 01:00:21.694287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.693745    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" event={"ID":"56a1068c-7f4a-4fed-94c1-d23f46614ea8","Type":"ContainerDied","Data":"1f1dc0f3a9f203dc42ddca28dd59fbe407f3ffdbd88bd8abf7ef2ce96e6f4c22"}
Nov 03 01:00:21.694287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.693846    6000 scope.go:122] "RemoveContainer" containerID="1c609e1bb1774bc6c0c3a0bb19108760b9029083772a98811f945cf2817856bc"
Nov 03 01:00:21.694440 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.694245    6000 scope.go:122] "RemoveContainer" containerID="1f1dc0f3a9f203dc42ddca28dd59fbe407f3ffdbd88bd8abf7ef2ce96e6f4c22"
Nov 03 01:00:21.694684 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:21.694653    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z_odh-ai-gateway-infra(56a1068c-7f4a-4fed-94c1-d23f46614ea8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8"
Nov 03 01:00:21.725877 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.725824    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"]
Nov 03 01:00:21.917699 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.917662    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:21.996406 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.996303    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-maas-api-service-ca\") pod \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") "
Nov 03 01:00:21.996672 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.996650    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-maas-api-service-ca" pod "b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" (UID: "b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:21.998427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.996432    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-kube-api-access-lntq4\" (UniqueName: \"kubernetes.io/projected/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-kube-api-access-lntq4\") pod \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\" (UID: \"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e\") "
Nov 03 01:00:21.999106 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.999055    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-kube-api-access-lntq4" pod "b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" (UID: "b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e"). InnerVolumeSpecName "kube-api-access-lntq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:21.999439 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.999422    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:21.999514 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:21.999449    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lntq4\" (UniqueName: \"kubernetes.io/projected/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e-kube-api-access-lntq4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:22.659041 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.653327    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" path="/var/lib/kubelet/pods/031c3046-9656-45e9-ae52-9fc2019b6cd2/volumes"
Nov 03 01:00:22.659041 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.654301    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" path="/var/lib/kubelet/pods/bd9ac712-8859-4943-8cc0-0056cdba1619/volumes"
Nov 03 01:00:22.659366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.659133    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" path="/var/lib/kubelet/pods/f32356b3-4afc-4427-b684-e23c7a7838fb/volumes"
Nov 03 01:00:22.708415 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.708383    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"
Nov 03 01:00:22.709045 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.708386    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd" event={"ID":"b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e","Type":"ContainerDied","Data":"6aec7b7405a27c5f66440ef66198591e012d32718cd0e51d88a5273b07dd75ee"}
Nov 03 01:00:22.709045 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.708538    6000 scope.go:122] "RemoveContainer" containerID="c21139eabec0e85e6a09fb3b827b9f62efda71424549cfdc9befd06b40afcdd1"
Nov 03 01:00:22.844162 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.844124    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"]
Nov 03 01:00:22.846727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.846698    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-955dc3-29894460-mzfnd"]
Nov 03 01:00:22.909654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:22.909590    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:23.075768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.075720    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56a1068c-7f4a-4fed-94c1-d23f46614ea8-kube-api-access-d9flw\" (UniqueName: \"kubernetes.io/projected/56a1068c-7f4a-4fed-94c1-d23f46614ea8-kube-api-access-d9flw\") pod \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") "
Nov 03 01:00:23.076072 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.076048    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56a1068c-7f4a-4fed-94c1-d23f46614ea8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a1068c-7f4a-4fed-94c1-d23f46614ea8-maas-api-service-ca\") pod \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\" (UID: \"56a1068c-7f4a-4fed-94c1-d23f46614ea8\") "
Nov 03 01:00:23.076600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.076572    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56a1068c-7f4a-4fed-94c1-d23f46614ea8-maas-api-service-ca" pod "56a1068c-7f4a-4fed-94c1-d23f46614ea8" (UID: "56a1068c-7f4a-4fed-94c1-d23f46614ea8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:23.077087 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.077062    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/56a1068c-7f4a-4fed-94c1-d23f46614ea8-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.078690 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.078664    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56a1068c-7f4a-4fed-94c1-d23f46614ea8-kube-api-access-d9flw" pod "56a1068c-7f4a-4fed-94c1-d23f46614ea8" (UID: "56a1068c-7f4a-4fed-94c1-d23f46614ea8"). InnerVolumeSpecName "kube-api-access-d9flw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:23.178707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.178608    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9flw\" (UniqueName: \"kubernetes.io/projected/56a1068c-7f4a-4fed-94c1-d23f46614ea8-kube-api-access-d9flw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:23.723510 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.723484    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"
Nov 03 01:00:23.724034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.723483    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z" event={"ID":"56a1068c-7f4a-4fed-94c1-d23f46614ea8","Type":"ContainerDied","Data":"dff319b8927a142575de0cb5937b197305f4e723f316b6edbfe948a656500917"}
Nov 03 01:00:23.724034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.723618    6000 scope.go:122] "RemoveContainer" containerID="1f1dc0f3a9f203dc42ddca28dd59fbe407f3ffdbd88bd8abf7ef2ce96e6f4c22"
Nov 03 01:00:23.773194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.773162    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"]
Nov 03 01:00:23.784113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:23.784073    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-c738e0-29894460-9bd7z"]
Nov 03 01:00:24.569901 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:24.569822    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" path="/var/lib/kubelet/pods/56a1068c-7f4a-4fed-94c1-d23f46614ea8/volumes"
Nov 03 01:00:24.570332 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:24.570314    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" path="/var/lib/kubelet/pods/b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e/volumes"
Nov 03 01:00:28.190921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:28.190883    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"]
Nov 03 01:00:30.440721 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:30.440682    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:33.191851 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.191780    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n" podUID="a5a89420-6064-470b-a029-d349ee66dff6" containerName="payload-processing" containerID="cri-o://fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d" gracePeriod=25
Nov 03 01:00:33.200910 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:33.200877    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:33.510439 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.510213    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 01:00:33.656004 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.655973    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5a89420-6064-470b-a029-d349ee66dff6-kube-api-access-sv9m8\" (UniqueName: \"kubernetes.io/projected/a5a89420-6064-470b-a029-d349ee66dff6-kube-api-access-sv9m8\") pod \"a5a89420-6064-470b-a029-d349ee66dff6\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") "
Nov 03 01:00:33.656179 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.656037    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5a89420-6064-470b-a029-d349ee66dff6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a89420-6064-470b-a029-d349ee66dff6-plugins-config-volume\") pod \"a5a89420-6064-470b-a029-d349ee66dff6\" (UID: \"a5a89420-6064-470b-a029-d349ee66dff6\") "
Nov 03 01:00:33.656549 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.656514    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5a89420-6064-470b-a029-d349ee66dff6-plugins-config-volume" pod "a5a89420-6064-470b-a029-d349ee66dff6" (UID: "a5a89420-6064-470b-a029-d349ee66dff6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:00:33.656949 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.656926    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5a89420-6064-470b-a029-d349ee66dff6-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:33.658245 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.658217    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5a89420-6064-470b-a029-d349ee66dff6-kube-api-access-sv9m8" pod "a5a89420-6064-470b-a029-d349ee66dff6" (UID: "a5a89420-6064-470b-a029-d349ee66dff6"). InnerVolumeSpecName "kube-api-access-sv9m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:00:33.758739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.758665    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sv9m8\" (UniqueName: \"kubernetes.io/projected/a5a89420-6064-470b-a029-d349ee66dff6-kube-api-access-sv9m8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:00:33.855796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.855763    6000 generic.go:356] "Generic (PLEG): container finished" podID="a5a89420-6064-470b-a029-d349ee66dff6" containerID="fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d" exitCode=0
Nov 03 01:00:33.855953 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.855816    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"
Nov 03 01:00:33.855953 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.855848    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n" event={"ID":"a5a89420-6064-470b-a029-d349ee66dff6","Type":"ContainerDied","Data":"fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d"}
Nov 03 01:00:33.855953 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.855900    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n" event={"ID":"a5a89420-6064-470b-a029-d349ee66dff6","Type":"ContainerDied","Data":"179d3a292d11400d7a304236950dadbc23632b41a3c565a4cb0097abe5bc609b"}
Nov 03 01:00:33.855953 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.855922    6000 scope.go:122] "RemoveContainer" containerID="fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d"
Nov 03 01:00:33.868572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.868551    6000 scope.go:122] "RemoveContainer" containerID="fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d"
Nov 03 01:00:33.868867 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:33.868847    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d\": container with ID starting with fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d not found: ID does not exist" containerID="fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d"
Nov 03 01:00:33.868954 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.868879    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d"} err="failed to get container status \"fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d\": rpc error: code = NotFound desc = could not find container \"fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d\": container with ID starting with fc8c53d0a13cb620dd297d4472c3bea6006a55e18eef81949ce07e784fe7af4d not found: ID does not exist"
Nov 03 01:00:33.907481 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.907441    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"]
Nov 03 01:00:33.911519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:33.911494    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-84f605-6d99b7fb8c-lqs2n"]
Nov 03 01:00:34.569764 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:34.569735    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5a89420-6064-470b-a029-d349ee66dff6" path="/var/lib/kubelet/pods/a5a89420-6064-470b-a029-d349ee66dff6/volumes"
Nov 03 01:00:40.564964 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:40.564931    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:43.847956 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.847921    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"]
Nov 03 01:00:43.849047 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849021    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:00:43.849047 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849047    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849069    6000 cpu_manager.go:414] "containerMap: removing container" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849077    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849099    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849108    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849125    6000 cpu_manager.go:414] "containerMap: removing container" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849131    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849146    6000 cpu_manager.go:414] "containerMap: removing container" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849153    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849164    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849171    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849184    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849190    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:00:43.849202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849203    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849210    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849275    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9163c109-b64a-4e78-943f-c8e73a20682b" containerName="main"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849283    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9163c109-b64a-4e78-943f-c8e73a20682b" containerName="main"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849300    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849305    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849312    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849317    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849325    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849330    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849337    6000 cpu_manager.go:414] "containerMap: removing container" podUID="a5a89420-6064-470b-a029-d349ee66dff6" containerName="payload-processing"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849341    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5a89420-6064-470b-a029-d349ee66dff6" containerName="payload-processing"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849354    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849358    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849364    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849369    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849378    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849384    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849395    6000 cpu_manager.go:414] "containerMap: removing container" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849399    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849553    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849562    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849570    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849576    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849583    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849590    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849596    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849602    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849608    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849616    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849621    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849632    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849637    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9163c109-b64a-4e78-943f-c8e73a20682b" containerName="main"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849645    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849651    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5a89420-6064-470b-a029-d349ee66dff6" containerName="payload-processing"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849656    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849661    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849667    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.849693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.849673    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8b1f0ed-acf8-4c9a-94a0-ce2f18d2aa0e" containerName="cleanup"
Nov 03 01:00:43.853583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.853567    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:43.879080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.879054    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-537652-serving-cert\""
Nov 03 01:00:43.883399 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.883375    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"]
Nov 03 01:00:43.953234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.953196    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28473950-3c2f-4dbc-be97-6c05b5efced2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:43.953421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.953349    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88pz\" (UniqueName: \"kubernetes.io/projected/28473950-3c2f-4dbc-be97-6c05b5efced2-kube-api-access-v88pz\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:43.953467 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.953441    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-maas-api-tls\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:43.953599 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:43.953582    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-metrics-certs\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.110554 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.107517    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28473950-3c2f-4dbc-be97-6c05b5efced2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.110554 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.107642    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88pz\" (UniqueName: \"kubernetes.io/projected/28473950-3c2f-4dbc-be97-6c05b5efced2-kube-api-access-v88pz\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.110554 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.107727    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-maas-api-tls\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.110554 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.107926    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-metrics-certs\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.110554 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.109995    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28473950-3c2f-4dbc-be97-6c05b5efced2-openshift-service-ca\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.129304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.128722    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-maas-api-tls\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.137448 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.137172    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-metrics-certs\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.137448 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.137433    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88pz\" (UniqueName: \"kubernetes.io/projected/28473950-3c2f-4dbc-be97-6c05b5efced2-kube-api-access-v88pz\") pod \"maas-api-e2e-inf-b-537652-6c68d697c9-hnq46\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.168822 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.168795    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:44.475086 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:00:44.475059    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28473950_3c2f_4dbc_be97_6c05b5efced2.slice/crio-24d8b9123a68cd5c4dea60a63f694658b11e498fc0a72f4df04f8ef3a19a03ba WatchSource:0}: Error finding container 24d8b9123a68cd5c4dea60a63f694658b11e498fc0a72f4df04f8ef3a19a03ba: Status 404 returned error can't find the container with id 24d8b9123a68cd5c4dea60a63f694658b11e498fc0a72f4df04f8ef3a19a03ba
Nov 03 01:00:44.478108 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.478080    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"]
Nov 03 01:00:44.956957 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:44.956920    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46" event={"ID":"28473950-3c2f-4dbc-be97-6c05b5efced2","Type":"ContainerStarted","Data":"24d8b9123a68cd5c4dea60a63f694658b11e498fc0a72f4df04f8ef3a19a03ba"}
Nov 03 01:00:45.785895 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:45.785862    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:00:45.969149 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:45.969094    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46" event={"ID":"28473950-3c2f-4dbc-be97-6c05b5efced2","Type":"ContainerStarted","Data":"15cbf560f5227e06d19cc8bdad49771dd2f787df82e109ec1cec877a345aa30b"}
Nov 03 01:00:45.969617 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:45.969192    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:00:46.011525 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:46.011468    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46" podStartSLOduration=2.6196265050000003 podStartE2EDuration="3.011452563s" totalImagesPullingTime="391.826058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:00:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:00:46.00762982 +0000 UTC m=+3283.562981856" watchObservedRunningTime="2026-11-03 01:00:46.011452563 +0000 UTC m=+3283.566804606"
Nov 03 01:00:46.582884 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:46.582830    6000 pod_container_manager_linux.go:218] "Failed to delete cgroup paths" cgroupName=["kubepods","besteffort","pod9163c109-b64a-4e78-943f-c8e73a20682b"] err="unable to destroy cgroup paths for cgroup [kubepods besteffort pod9163c109-b64a-4e78-943f-c8e73a20682b] : Timed out while waiting for systemd to remove kubepods-besteffort-pod9163c109_b64a_4e78_943f_c8e73a20682b.slice"
Nov 03 01:00:46.582884 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:46.582882    6000 kubelet_pods.go:2797] "Failed to destroy orphaned pod cgroup" podUID="9163c109-b64a-4e78-943f-c8e73a20682b" err="failed to delete cgroup paths for [kubepods besteffort pod9163c109-b64a-4e78-943f-c8e73a20682b] : unable to destroy cgroup paths for cgroup [kubepods besteffort pod9163c109-b64a-4e78-943f-c8e73a20682b] : Timed out while waiting for systemd to remove kubepods-besteffort-pod9163c109_b64a_4e78_943f_c8e73a20682b.slice"
Nov 03 01:00:47.873412 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:47.873377    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:47.874086 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:47.873531    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:48.198500 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:48.198413    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:49.897735 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:49.897694    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:00:50.585666 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:00:50.585627    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:00:52.056925 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:00:52.056896    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:01:00.603851 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:01:00.603810    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf32356b3_4afc_4427_b684_e23c7a7838fb.slice/crio-conmon-bf458fe02451492de5af6379204ac138d2a2537a2f7a459b48676ce90a9abb1b.scope\": RecentStats: unable to find data in memory cache]"
Nov 03 01:01:11.110800 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.110767    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"]
Nov 03 01:01:11.111395 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.110997    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc" podUID="b411ea47-f364-4198-a54a-202168abadc6" containerName="maas-api" containerID="cri-o://3399cadd7194e827edacb9ad3121cf2121aa53241af165d65c1f370dbb72c9a6" gracePeriod=30
Nov 03 01:01:11.315178 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.315011    6000 generic.go:356] "Generic (PLEG): container finished" podID="b411ea47-f364-4198-a54a-202168abadc6" containerID="3399cadd7194e827edacb9ad3121cf2121aa53241af165d65c1f370dbb72c9a6" exitCode=0
Nov 03 01:01:11.315475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.315324    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc" event={"ID":"b411ea47-f364-4198-a54a-202168abadc6","Type":"ContainerDied","Data":"3399cadd7194e827edacb9ad3121cf2121aa53241af165d65c1f370dbb72c9a6"}
Nov 03 01:01:11.637277 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.636629    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 01:01:11.743473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.727899    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-maas-api-tls\") pod \"b411ea47-f364-4198-a54a-202168abadc6\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") "
Nov 03 01:01:11.743473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.728103    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b411ea47-f364-4198-a54a-202168abadc6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b411ea47-f364-4198-a54a-202168abadc6-openshift-service-ca\") pod \"b411ea47-f364-4198-a54a-202168abadc6\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") "
Nov 03 01:01:11.743473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.728359    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-metrics-certs\") pod \"b411ea47-f364-4198-a54a-202168abadc6\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") "
Nov 03 01:01:11.743473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.728448    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b411ea47-f364-4198-a54a-202168abadc6-kube-api-access-kv94h\" (UniqueName: \"kubernetes.io/projected/b411ea47-f364-4198-a54a-202168abadc6-kube-api-access-kv94h\") pod \"b411ea47-f364-4198-a54a-202168abadc6\" (UID: \"b411ea47-f364-4198-a54a-202168abadc6\") "
Nov 03 01:01:11.771329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.753729    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-maas-api-tls" pod "b411ea47-f364-4198-a54a-202168abadc6" (UID: "b411ea47-f364-4198-a54a-202168abadc6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:01:11.794311 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.777451    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b411ea47-f364-4198-a54a-202168abadc6-openshift-service-ca" pod "b411ea47-f364-4198-a54a-202168abadc6" (UID: "b411ea47-f364-4198-a54a-202168abadc6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:11.813844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.812453    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-metrics-certs" pod "b411ea47-f364-4198-a54a-202168abadc6" (UID: "b411ea47-f364-4198-a54a-202168abadc6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:01:11.816769 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.816726    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b411ea47-f364-4198-a54a-202168abadc6-kube-api-access-kv94h" pod "b411ea47-f364-4198-a54a-202168abadc6" (UID: "b411ea47-f364-4198-a54a-202168abadc6"). InnerVolumeSpecName "kube-api-access-kv94h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:11.831010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.830977    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b411ea47-f364-4198-a54a-202168abadc6-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:11.831010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.831009    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:11.831294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.831018    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv94h\" (UniqueName: \"kubernetes.io/projected/b411ea47-f364-4198-a54a-202168abadc6-kube-api-access-kv94h\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:11.831294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:11.831028    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b411ea47-f364-4198-a54a-202168abadc6-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:12.326010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:12.325977    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"
Nov 03 01:01:12.326530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:12.326012    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc" event={"ID":"b411ea47-f364-4198-a54a-202168abadc6","Type":"ContainerDied","Data":"8667af8a6b7dc43be66a7f55f465668ab773b5a87c58927e522e9caf6bf96c67"}
Nov 03 01:01:12.326530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:12.326071    6000 scope.go:122] "RemoveContainer" containerID="3399cadd7194e827edacb9ad3121cf2121aa53241af165d65c1f370dbb72c9a6"
Nov 03 01:01:12.385561 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:12.385471    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"]
Nov 03 01:01:12.391907 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:12.391875    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-84f605-6795b5f489-pn5jc"]
Nov 03 01:01:12.572385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:12.572347    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b411ea47-f364-4198-a54a-202168abadc6" path="/var/lib/kubelet/pods/b411ea47-f364-4198-a54a-202168abadc6/volumes"
Nov 03 01:01:16.803298 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:16.802674    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"]
Nov 03 01:01:16.803298 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:16.802949    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy" containerID="cri-o://631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004" gracePeriod=30
Nov 03 01:01:19.955871 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:01:19.955824    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:01:22.101120 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.101092    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 01:01:22.166459 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166381    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca3db227-6b69-4447-b265-ff64e57fc88f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca3db227-6b69-4447-b265-ff64e57fc88f-istiod-ca-cert\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.166459 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166438    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-token\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-token\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.166708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166485    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-socket\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.166708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166519    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-credential-socket\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.166708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166611    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-envoy\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.166900 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166785    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-kube-api-access-mcqsv\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-kube-api-access-mcqsv\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.166900 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166798    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca3db227-6b69-4447-b265-ff64e57fc88f-istiod-ca-cert" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:22.166900 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.166865    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-podinfo\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.167075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.167042    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-data\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.167075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.167056    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-credential-socket" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:22.167209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.167190    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-certs\") pod \"ca3db227-6b69-4447-b265-ff64e57fc88f\" (UID: \"ca3db227-6b69-4447-b265-ff64e57fc88f\") "
Nov 03 01:01:22.167295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.167221    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-socket" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:22.167707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.167680    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-data" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:22.167917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.167894    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-certs" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:22.168529 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.168504    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.168632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.168534    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.168632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.168548    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.168632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.168563    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.168632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.168575    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca3db227-6b69-4447-b265-ff64e57fc88f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.170410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.170376    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-token" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:22.170717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.170679    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-kube-api-access-mcqsv" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "kube-api-access-mcqsv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:22.172996 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.172970    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-envoy" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:01:22.173130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.173054    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-podinfo" pod "ca3db227-6b69-4447-b265-ff64e57fc88f" (UID: "ca3db227-6b69-4447-b265-ff64e57fc88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:01:22.270187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.270155    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.270187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.270182    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.270187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.270193    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcqsv\" (UniqueName: \"kubernetes.io/projected/ca3db227-6b69-4447-b265-ff64e57fc88f-kube-api-access-mcqsv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.270483 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.270204    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca3db227-6b69-4447-b265-ff64e57fc88f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:22.536567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.536484    6000 generic.go:356] "Generic (PLEG): container finished" podID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerID="631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004" exitCode=0
Nov 03 01:01:22.536567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.536534    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"
Nov 03 01:01:22.536767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.536575    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" event={"ID":"ca3db227-6b69-4447-b265-ff64e57fc88f","Type":"ContainerDied","Data":"631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004"}
Nov 03 01:01:22.536767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.536660    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn" event={"ID":"ca3db227-6b69-4447-b265-ff64e57fc88f","Type":"ContainerDied","Data":"4b58b76ec5fc7f565ca8dc808ba3897f7f42ad8e4913b09a8cf75bb0629a6d7d"}
Nov 03 01:01:22.536767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.536687    6000 scope.go:122] "RemoveContainer" containerID="631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004"
Nov 03 01:01:22.549467 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.549448    6000 scope.go:122] "RemoveContainer" containerID="b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c"
Nov 03 01:01:22.560179 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.560159    6000 scope.go:122] "RemoveContainer" containerID="631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004"
Nov 03 01:01:22.560494 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:01:22.560471    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004\": container with ID starting with 631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004 not found: ID does not exist" containerID="631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004"
Nov 03 01:01:22.560589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.560505    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004"} err="failed to get container status \"631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004\": rpc error: code = NotFound desc = could not find container \"631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004\": container with ID starting with 631dd7473cdbbf3a2d0010ab887fee4747fe6d5115aab7d2b05ee360b4e2a004 not found: ID does not exist"
Nov 03 01:01:22.560589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.560528    6000 scope.go:122] "RemoveContainer" containerID="b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c"
Nov 03 01:01:22.560779 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:01:22.560762    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c\": container with ID starting with b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c not found: ID does not exist" containerID="b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c"
Nov 03 01:01:22.560817 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.560785    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c"} err="failed to get container status \"b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c\": rpc error: code = NotFound desc = could not find container \"b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c\": container with ID starting with b69aee862539edd9a298c375f549eeaec87091d323bdcb3bc7bf22d052ae984c not found: ID does not exist"
Nov 03 01:01:22.592397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.592375    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"]
Nov 03 01:01:22.604964 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.604938    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-84f605-openshift-default-74967b9b5-tccjn"]
Nov 03 01:01:22.699722 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.699690    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"]
Nov 03 01:01:22.728496 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:22.728464    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"]
Nov 03 01:01:24.569889 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:24.569812    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" path="/var/lib/kubelet/pods/ca3db227-6b69-4447-b265-ff64e57fc88f/volumes"
Nov 03 01:01:24.857393 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:24.857358    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"]
Nov 03 01:01:24.857611 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:24.857581    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd" podUID="2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" containerName="maas-api" containerID="cri-o://0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0" gracePeriod=30
Nov 03 01:01:25.490551 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.490519    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 01:01:25.521156 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.521128    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-metrics-certs\") pod \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") "
Nov 03 01:01:25.521879 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.521856    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-openshift-service-ca\") pod \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") "
Nov 03 01:01:25.522618 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.522588    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls\") pod \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") "
Nov 03 01:01:25.523534 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.522362    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-openshift-service-ca" pod "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" (UID: "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:25.524602 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.524575    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-metrics-certs" pod "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" (UID: "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:01:25.526852 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.526831    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls" pod "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" (UID: "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:01:25.625200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.625119    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-kube-api-access-stkc9\" (UniqueName: \"kubernetes.io/projected/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-kube-api-access-stkc9\") pod \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\" (UID: \"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d\") "
Nov 03 01:01:25.625969 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.625946    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:25.626089 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.625975    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:25.626089 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.625990    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:25.628139 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.628112    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-kube-api-access-stkc9" pod "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" (UID: "2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d"). InnerVolumeSpecName "kube-api-access-stkc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:25.629172 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.629145    6000 generic.go:356] "Generic (PLEG): container finished" podID="2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" containerID="0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0" exitCode=0
Nov 03 01:01:25.629322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.629193    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd" event={"ID":"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d","Type":"ContainerDied","Data":"0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0"}
Nov 03 01:01:25.629322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.629207    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"
Nov 03 01:01:25.629322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.629246    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd" event={"ID":"2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d","Type":"ContainerDied","Data":"fc4bc65e3153c751b64407e30a0f4cc246b6f513cb7de677ec75436839544525"}
Nov 03 01:01:25.629322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.629293    6000 scope.go:122] "RemoveContainer" containerID="0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0"
Nov 03 01:01:25.650130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.646604    6000 scope.go:122] "RemoveContainer" containerID="0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0"
Nov 03 01:01:25.650130 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:01:25.647062    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0\": container with ID starting with 0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0 not found: ID does not exist" containerID="0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0"
Nov 03 01:01:25.650130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.647099    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0"} err="failed to get container status \"0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0\": rpc error: code = NotFound desc = could not find container \"0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0\": container with ID starting with 0e6c3bc3af1a2c44e113c801516d811c1cc9c58d758a417f42ea46cb5d82c7c0 not found: ID does not exist"
Nov 03 01:01:25.684499 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.684464    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"]
Nov 03 01:01:25.688313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.688270    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-726205-6d67b7849f-scbgd"]
Nov 03 01:01:25.728135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:25.728105    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stkc9\" (UniqueName: \"kubernetes.io/projected/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d-kube-api-access-stkc9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:26.569703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:26.569673    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" path="/var/lib/kubelet/pods/2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d/volumes"
Nov 03 01:01:27.700159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:27.700107    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" podUID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" containerName="payload-processing" containerID="cri-o://7556a21b1c2c9200956f0fe7059ae0ab352701b7723f3422c92d29c30d197dd4" gracePeriod=25
Nov 03 01:01:27.728869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:27.728835    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" podUID="2d27238b-448c-44e1-b684-fee5f43d838c" containerName="payload-pre-processing" containerID="cri-o://3cab8aadb71151633141c107fa71a8c5b607fff31f0e672f019eb28d91fa23cc" gracePeriod=25
Nov 03 01:01:37.268123 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:37.268089    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" podUID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:01:37.268839 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:37.268096    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" podUID="2d27238b-448c-44e1-b684-fee5f43d838c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:01:47.268844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:47.268760    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" podUID="2d27238b-448c-44e1-b684-fee5f43d838c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:01:47.269291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:47.268767    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" podUID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:01:47.880328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:47.880092    6000 generic.go:356] "Generic (PLEG): container finished" podID="2d27238b-448c-44e1-b684-fee5f43d838c" containerID="3cab8aadb71151633141c107fa71a8c5b607fff31f0e672f019eb28d91fa23cc" exitCode=0
Nov 03 01:01:47.880328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:47.880184    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" event={"ID":"2d27238b-448c-44e1-b684-fee5f43d838c","Type":"ContainerDied","Data":"3cab8aadb71151633141c107fa71a8c5b607fff31f0e672f019eb28d91fa23cc"}
Nov 03 01:01:47.883330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:47.883301    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" event={"ID":"48bd335d-7461-4bb4-9f26-f9a4f22d9454","Type":"ContainerDied","Data":"7556a21b1c2c9200956f0fe7059ae0ab352701b7723f3422c92d29c30d197dd4"}
Nov 03 01:01:47.883506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:47.883368    6000 generic.go:356] "Generic (PLEG): container finished" podID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" containerID="7556a21b1c2c9200956f0fe7059ae0ab352701b7723f3422c92d29c30d197dd4" exitCode=0
Nov 03 01:01:48.052958 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.052926    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 01:01:48.086614 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.086586    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 01:01:48.145643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.145610    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48bd335d-7461-4bb4-9f26-f9a4f22d9454-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd335d-7461-4bb4-9f26-f9a4f22d9454-plugins-config-volume\") pod \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") "
Nov 03 01:01:48.145855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.145714    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48bd335d-7461-4bb4-9f26-f9a4f22d9454-kube-api-access-sm6s4\" (UniqueName: \"kubernetes.io/projected/48bd335d-7461-4bb4-9f26-f9a4f22d9454-kube-api-access-sm6s4\") pod \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\" (UID: \"48bd335d-7461-4bb4-9f26-f9a4f22d9454\") "
Nov 03 01:01:48.146006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.145983    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48bd335d-7461-4bb4-9f26-f9a4f22d9454-plugins-config-volume" pod "48bd335d-7461-4bb4-9f26-f9a4f22d9454" (UID: "48bd335d-7461-4bb4-9f26-f9a4f22d9454"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:48.146516 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.146491    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/48bd335d-7461-4bb4-9f26-f9a4f22d9454-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:48.148056 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.148035    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48bd335d-7461-4bb4-9f26-f9a4f22d9454-kube-api-access-sm6s4" pod "48bd335d-7461-4bb4-9f26-f9a4f22d9454" (UID: "48bd335d-7461-4bb4-9f26-f9a4f22d9454"). InnerVolumeSpecName "kube-api-access-sm6s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:48.247736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.247705    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d27238b-448c-44e1-b684-fee5f43d838c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d27238b-448c-44e1-b684-fee5f43d838c-plugins-config-volume\") pod \"2d27238b-448c-44e1-b684-fee5f43d838c\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") "
Nov 03 01:01:48.247924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.247782    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d27238b-448c-44e1-b684-fee5f43d838c-kube-api-access-f95k7\" (UniqueName: \"kubernetes.io/projected/2d27238b-448c-44e1-b684-fee5f43d838c-kube-api-access-f95k7\") pod \"2d27238b-448c-44e1-b684-fee5f43d838c\" (UID: \"2d27238b-448c-44e1-b684-fee5f43d838c\") "
Nov 03 01:01:48.248095 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.248073    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d27238b-448c-44e1-b684-fee5f43d838c-plugins-config-volume" pod "2d27238b-448c-44e1-b684-fee5f43d838c" (UID: "2d27238b-448c-44e1-b684-fee5f43d838c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:01:48.248239 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.248224    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm6s4\" (UniqueName: \"kubernetes.io/projected/48bd335d-7461-4bb4-9f26-f9a4f22d9454-kube-api-access-sm6s4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:48.248298 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.248244    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d27238b-448c-44e1-b684-fee5f43d838c-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:48.250133 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.250108    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d27238b-448c-44e1-b684-fee5f43d838c-kube-api-access-f95k7" pod "2d27238b-448c-44e1-b684-fee5f43d838c" (UID: "2d27238b-448c-44e1-b684-fee5f43d838c"). InnerVolumeSpecName "kube-api-access-f95k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:01:48.349454 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.349368    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f95k7\" (UniqueName: \"kubernetes.io/projected/2d27238b-448c-44e1-b684-fee5f43d838c-kube-api-access-f95k7\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:01:48.892543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.892511    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"
Nov 03 01:01:48.892739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.892513    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf" event={"ID":"2d27238b-448c-44e1-b684-fee5f43d838c","Type":"ContainerDied","Data":"dd114b3f24a59174dfa0ea6b7c791f7326731fc516b175ffb22826e0cda54e8f"}
Nov 03 01:01:48.892739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.892658    6000 scope.go:122] "RemoveContainer" containerID="3cab8aadb71151633141c107fa71a8c5b607fff31f0e672f019eb28d91fa23cc"
Nov 03 01:01:48.894174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.894142    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"
Nov 03 01:01:48.894277 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.894186    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s" event={"ID":"48bd335d-7461-4bb4-9f26-f9a4f22d9454","Type":"ContainerDied","Data":"4671ef5143a4953c8924211fe719874d7da54d9a432f477c82642b207a9fb3a2"}
Nov 03 01:01:48.904159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.904139    6000 scope.go:122] "RemoveContainer" containerID="7556a21b1c2c9200956f0fe7059ae0ab352701b7723f3422c92d29c30d197dd4"
Nov 03 01:01:48.946284 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.946225    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"]
Nov 03 01:01:48.957412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.957387    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-726205-845cc5ff87-zgcrf"]
Nov 03 01:01:48.980807 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.980607    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"]
Nov 03 01:01:48.984513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:48.984492    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-726205-694cb8fcff-tdh6s"]
Nov 03 01:01:49.902507 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:01:49.902471    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:01:50.570691 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:50.570663    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d27238b-448c-44e1-b684-fee5f43d838c" path="/var/lib/kubelet/pods/2d27238b-448c-44e1-b684-fee5f43d838c/volumes"
Nov 03 01:01:50.571153 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:01:50.571138    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" path="/var/lib/kubelet/pods/48bd335d-7461-4bb4-9f26-f9a4f22d9454/volumes"
Nov 03 01:02:07.481584 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:07.474943    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:02:11.946394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:11.946354    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dqqvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:02:11.947134 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:11.947101    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:02:19.898272 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:02:19.898217    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:02:22.160690 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.160655    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"]
Nov 03 01:02:22.162134 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162093    6000 cpu_manager.go:414] "containerMap: removing container" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:02:22.162134 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162119    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="031c3046-9656-45e9-ae52-9fc2019b6cd2" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162143    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162151    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162161    6000 cpu_manager.go:414] "containerMap: removing container" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162167    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162184    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162190    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd9ac712-8859-4943-8cc0-0056cdba1619" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162209    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162215    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162229    6000 cpu_manager.go:414] "containerMap: removing container" podUID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" containerName="payload-processing"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162236    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" containerName="payload-processing"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162306    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2d27238b-448c-44e1-b684-fee5f43d838c" containerName="payload-pre-processing"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162316    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d27238b-448c-44e1-b684-fee5f43d838c" containerName="payload-pre-processing"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162332    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162339    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162354    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162360    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162375    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162382    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162416    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b411ea47-f364-4198-a54a-202168abadc6" containerName="maas-api"
Nov 03 01:02:22.162429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162423    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b411ea47-f364-4198-a54a-202168abadc6" containerName="maas-api"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162467    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" containerName="maas-api"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162475    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" containerName="maas-api"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162713    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d27238b-448c-44e1-b684-fee5f43d838c" containerName="payload-pre-processing"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162730    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f32356b3-4afc-4427-b684-e23c7a7838fb" containerName="cleanup"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162743    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162759    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b411ea47-f364-4198-a54a-202168abadc6" containerName="maas-api"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162772    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2e711e9-1280-4d26-b552-4a34bb3544b3" containerName="cleanup"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162784    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c3f95e5-2a71-4a6f-8f09-0f342b84ce6d" containerName="maas-api"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162796    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca3db227-6b69-4447-b265-ff64e57fc88f" containerName="istio-proxy"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162803    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="56a1068c-7f4a-4fed-94c1-d23f46614ea8" containerName="cleanup"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162816    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b7b142-012c-4838-81a7-2404a362caa6" containerName="cleanup"
Nov 03 01:02:22.163462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.162823    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="48bd335d-7461-4bb4-9f26-f9a4f22d9454" containerName="payload-processing"
Nov 03 01:02:22.169106 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.169080    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.195063 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.195033    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"]
Nov 03 01:02:22.197411 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.197383    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8ed55a\"/\"openshift-service-ca.crt\""
Nov 03 01:02:22.197570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.197455    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8ed55a\"/\"default-dockercfg-l4dmv\""
Nov 03 01:02:22.197570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.197526    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-8ed55a\"/\"test-model-8ed55a-kserve-self-signed-certs\""
Nov 03 01:02:22.197570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.197501    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-8ed55a\"/\"kube-root-ca.crt\""
Nov 03 01:02:22.298137 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.298102    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tmp-dir\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.298345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.298222    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-dshm\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.298345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.298301    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg4fk\" (UniqueName: \"kubernetes.io/projected/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-kube-api-access-jg4fk\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.298563 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.298540    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tls-certs\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.298602 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.298590    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-home\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.298741 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.298725    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-model-cache\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.400450    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-dshm\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.400530    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg4fk\" (UniqueName: \"kubernetes.io/projected/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-kube-api-access-jg4fk\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.400723    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tls-certs\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.400764    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-home\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.400841    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-model-cache\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.400976    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tmp-dir\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.401462    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tmp-dir\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.402361    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-home\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.422327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.402632    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-model-cache\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.467279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.459164    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tls-certs\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.467279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.465719    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg4fk\" (UniqueName: \"kubernetes.io/projected/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-kube-api-access-jg4fk\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.490281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.487532    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-dshm\") pod \"test-model-8ed55a-kserve-86774654c5-r96l9\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") " pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.497574 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.496761    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:22.870462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:22.870333    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"]
Nov 03 01:02:22.873459 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:02:22.873428    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a83d623_1d5f_4dfa_8b5a_f826ec572c44.slice/crio-d3e18954cbea65f14326209bde758bc3a66b7cd095009099428f41a3f4152287 WatchSource:0}: Error finding container d3e18954cbea65f14326209bde758bc3a66b7cd095009099428f41a3f4152287: Status 404 returned error can't find the container with id d3e18954cbea65f14326209bde758bc3a66b7cd095009099428f41a3f4152287
Nov 03 01:02:23.275951 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:23.275837    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9" event={"ID":"6a83d623-1d5f-4dfa-8b5a-f826ec572c44","Type":"ContainerStarted","Data":"26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526"}
Nov 03 01:02:23.275951 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:23.275896    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9" event={"ID":"6a83d623-1d5f-4dfa-8b5a-f826ec572c44","Type":"ContainerStarted","Data":"d3e18954cbea65f14326209bde758bc3a66b7cd095009099428f41a3f4152287"}
Nov 03 01:02:23.318124 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:23.318065    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9" podStartSLOduration=1.318052134 podStartE2EDuration="1.318052134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:23.317427774 +0000 UTC m=+3380.872779799" watchObservedRunningTime="2026-11-03 01:02:23.318052134 +0000 UTC m=+3380.873404158"
Nov 03 01:02:23.497978 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:23.497949    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:25.388753 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.388718    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"]
Nov 03 01:02:25.395597 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.395572    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.414446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.414418    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"]
Nov 03 01:02:25.417816 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.417788    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-70c7d6\"/\"openshift-service-ca.crt\""
Nov 03 01:02:25.418015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.417891    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-70c7d6\"/\"test-model-70c7d6-kserve-self-signed-certs\""
Nov 03 01:02:25.423055 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.422473    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-70c7d6\"/\"default-dockercfg-wrdd9\""
Nov 03 01:02:25.423055 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.422594    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-70c7d6\"/\"kube-root-ca.crt\""
Nov 03 01:02:25.492320 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.492238    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-dshm\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.492509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.492366    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtp79\" (UniqueName: \"kubernetes.io/projected/b703fed3-ecc3-44fc-9275-0831d71e9e1c-kube-api-access-jtp79\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.492509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.492469    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-model-cache\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.492725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.492695    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-home\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.492850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.492761    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tls-certs\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.492898 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.492868    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tmp-dir\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.593949 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.593911    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-home\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.594113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.593970    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tls-certs\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.594113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.594098    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tmp-dir\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.594363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.594336    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-dshm\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.594427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.594381    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtp79\" (UniqueName: \"kubernetes.io/projected/b703fed3-ecc3-44fc-9275-0831d71e9e1c-kube-api-access-jtp79\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.594427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.594391    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-home\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.594427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.594418    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tmp-dir\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.594538 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.594479    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-model-cache\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.595135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.595107    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-model-cache\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.596857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.596829    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-dshm\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.596857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.596842    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tls-certs\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.641504 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.641360    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtp79\" (UniqueName: \"kubernetes.io/projected/b703fed3-ecc3-44fc-9275-0831d71e9e1c-kube-api-access-jtp79\") pod \"test-model-70c7d6-kserve-69486c8fc7-8smqn\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") " pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.691300 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.687449    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"]
Nov 03 01:02:25.697359 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.696318    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.757921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.746420    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:25.812490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.812156    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.813665 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.813637    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.813995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.813976    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.814221 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.814203    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvv9t\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-kube-api-access-xvv9t\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.856780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.821311    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"]
Nov 03 01:02:25.856780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.842905    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.856780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.843143    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.856780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.843191    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e4b26-c21e-4fde-b787-33bb4685879b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.856780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.843557    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.856780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.843863    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.956999    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957049    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvv9t\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-kube-api-access-xvv9t\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957079    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957100    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957122    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e4b26-c21e-4fde-b787-33bb4685879b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957170    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957234    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957306    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957439    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.957957 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.957929    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.958185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.958166    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.959860 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.959194    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.960072 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.960048    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.960072 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.960066    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e4b26-c21e-4fde-b787-33bb4685879b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.962344 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.962320    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.963235 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.963212    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.980743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.980710    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvv9t\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-kube-api-access-xvv9t\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:25.981960 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:25.981936    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-666b4\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:26.112384 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.112345    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:26.197323 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.197215    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"]
Nov 03 01:02:26.207117 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:02:26.207080    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb703fed3_ecc3_44fc_9275_0831d71e9e1c.slice/crio-3d09c1f52f2ca4a13e59e57616528b9760ba6b0597175cc30bb3b6981106b592 WatchSource:0}: Error finding container 3d09c1f52f2ca4a13e59e57616528b9760ba6b0597175cc30bb3b6981106b592: Status 404 returned error can't find the container with id 3d09c1f52f2ca4a13e59e57616528b9760ba6b0597175cc30bb3b6981106b592
Nov 03 01:02:26.303855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.303813    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn" event={"ID":"b703fed3-ecc3-44fc-9275-0831d71e9e1c","Type":"ContainerStarted","Data":"74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15"}
Nov 03 01:02:26.304053 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.303884    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn" event={"ID":"b703fed3-ecc3-44fc-9275-0831d71e9e1c","Type":"ContainerStarted","Data":"3d09c1f52f2ca4a13e59e57616528b9760ba6b0597175cc30bb3b6981106b592"}
Nov 03 01:02:26.347735 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.347676    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn" podStartSLOduration=1.347662668 podStartE2EDuration="1.347662668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:26.342202062 +0000 UTC m=+3383.897554109" watchObservedRunningTime="2026-11-03 01:02:26.347662668 +0000 UTC m=+3383.903014691"
Nov 03 01:02:26.423042 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:02:26.423015    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b5e4b26_c21e_4fde_b787_33bb4685879b.slice/crio-0cdc0918152a6e75926d890bca39a2bd241c914b5ce32a8a022bf28d8590d638 WatchSource:0}: Error finding container 0cdc0918152a6e75926d890bca39a2bd241c914b5ce32a8a022bf28d8590d638: Status 404 returned error can't find the container with id 0cdc0918152a6e75926d890bca39a2bd241c914b5ce32a8a022bf28d8590d638
Nov 03 01:02:26.423905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.423880    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"]
Nov 03 01:02:26.426318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.426246    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:02:26.426409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.426357    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:02:26.426409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.426388    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:02:26.799839 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:26.799758    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:27.321480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:27.321444    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" event={"ID":"2b5e4b26-c21e-4fde-b787-33bb4685879b","Type":"ContainerStarted","Data":"0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5"}
Nov 03 01:02:27.321676 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:27.321501    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" event={"ID":"2b5e4b26-c21e-4fde-b787-33bb4685879b","Type":"ContainerStarted","Data":"0cdc0918152a6e75926d890bca39a2bd241c914b5ce32a8a022bf28d8590d638"}
Nov 03 01:02:27.364381 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:27.364319    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podStartSLOduration=2.36429922 podStartE2EDuration="2.36429922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:02:27.359912562 +0000 UTC m=+3384.915264587" watchObservedRunningTime="2026-11-03 01:02:27.36429922 +0000 UTC m=+3384.919651245"
Nov 03 01:02:28.112667 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:28.112627    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:28.115434 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:28.115406    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-666b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 03 01:02:28.115601 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:28.115463    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 03 01:02:29.113115 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:29.113082    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-666b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 03 01:02:29.113575 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:29.113135    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 03 01:02:30.112702 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:30.112665    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-666b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 03 01:02:30.112905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:30.112719    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 03 01:02:31.112749 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:31.112711    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-666b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 03 01:02:31.113175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:31.112771    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 03 01:02:32.112732 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:32.112697    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-666b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 03 01:02:32.112955 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:32.112754    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 03 01:02:32.501772 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:32.501679    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:32.523789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:32.523760    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:33.113361 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:33.113318    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-666b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 03 01:02:33.113908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:33.113375    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 03 01:02:33.453579 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:33.453489    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:02:34.112936 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:34.112904    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-666b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused" start-of-body=
Nov 03 01:02:34.113120 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:34.112961    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.28:15021/healthz/ready\": dial tcp 10.134.1.28:15021: connect: connection refused"
Nov 03 01:02:35.518087 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:35.518035    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:35.519006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:35.518978    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:35.519620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:35.519599    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:02:35.800649 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:35.800566    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:35.813105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:35.813079    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:36.486641 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:36.486613    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:02:49.897976 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:02:49.897929    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:02:52.440524 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:52.440479    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"]
Nov 03 01:02:52.441054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:52.440713    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy" containerID="cri-o://b45e01494524204cec11cb1362db77f3e2d0efe0b86848fb09c50dcaa9657859" gracePeriod=30
Nov 03 01:02:56.877993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:56.877949    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:02:56.878751 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:56.878723    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:02:57.690416 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.690384    6000 generic.go:356] "Generic (PLEG): container finished" podID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerID="b45e01494524204cec11cb1362db77f3e2d0efe0b86848fb09c50dcaa9657859" exitCode=0
Nov 03 01:02:57.690621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.690462    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" event={"ID":"a5f12ebd-dce0-4c4e-920b-c260acdb3b17","Type":"ContainerDied","Data":"b45e01494524204cec11cb1362db77f3e2d0efe0b86848fb09c50dcaa9657859"}
Nov 03 01:02:57.740687 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.740658    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 01:02:57.901998 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.901967    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-envoy\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902221    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-certs\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902296    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istiod-ca-cert\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902352    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-credential-socket\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902416    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-data\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902723 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902498    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-kube-api-access-hz9wl\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-kube-api-access-hz9wl\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902723 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902555    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-certs" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:02:57.902723 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902581    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-token\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-token\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902723 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902680    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-credential-socket" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:02:57.902723 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902700    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-podinfo\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.902993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902741    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istiod-ca-cert" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:02:57.902993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902750    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-socket\") pod \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\" (UID: \"a5f12ebd-dce0-4c4e-920b-c260acdb3b17\") "
Nov 03 01:02:57.903100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.902984    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-data" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:02:57.903100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.903087    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-socket" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:02:57.903594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.903573    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:57.903717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.903597    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:57.903717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.903609    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:57.903717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.903623    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:57.903717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.903634    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:57.904475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.904448    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-envoy" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:02:57.904833 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.904812    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-podinfo" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:02:57.905385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.905361    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-kube-api-access-hz9wl" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "kube-api-access-hz9wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:02:57.905493 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:57.905390    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-token" pod "a5f12ebd-dce0-4c4e-920b-c260acdb3b17" (UID: "a5f12ebd-dce0-4c4e-920b-c260acdb3b17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:02:58.005540 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.005453    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:58.005540 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.005483    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz9wl\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-kube-api-access-hz9wl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:58.005540 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.005492    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:58.005540 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.005501    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a5f12ebd-dce0-4c4e-920b-c260acdb3b17-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:02:58.700084 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.700054    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"
Nov 03 01:02:58.700302 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.700063    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn" event={"ID":"a5f12ebd-dce0-4c4e-920b-c260acdb3b17","Type":"ContainerDied","Data":"46d9901db10d4fbe2893fd5a95f4eda33fb3aa1b17d409a3de4161e163a34adc"}
Nov 03 01:02:58.700302 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.700190    6000 scope.go:122] "RemoveContainer" containerID="b45e01494524204cec11cb1362db77f3e2d0efe0b86848fb09c50dcaa9657859"
Nov 03 01:02:58.762376 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.762343    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"]
Nov 03 01:02:58.773356 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:02:58.773328    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-fdacdb-openshift-default-5955cd97bf-gbfmn"]
Nov 03 01:03:00.570246 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:00.570214    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" path="/var/lib/kubelet/pods/a5f12ebd-dce0-4c4e-920b-c260acdb3b17/volumes"
Nov 03 01:03:13.728432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:13.728332    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"]
Nov 03 01:03:13.728942 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:13.728567    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy" containerID="cri-o://30cc1f9cc0cbf0dca64e56b307584c7db06fc0fc5a1e1d38c19029e24d90c91d" gracePeriod=30
Nov 03 01:03:16.638887 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:16.638835    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"]
Nov 03 01:03:16.639482 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:16.639133    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" containerID="cri-o://ab91dd6abce2634b2b283838312f4a316177327ae531ae4ce3fbd67f7da84486" gracePeriod=30
Nov 03 01:03:18.324013 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:18.323975    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:03:18.740717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:18.740677    6000 patch_prober.go:29] interesting pod/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.218:15021/healthz/ready\": read tcp 10.134.0.2:53502->10.134.0.218:15021: read: connection reset by peer" start-of-body=
Nov 03 01:03:18.740896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:18.740746    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.218:15021/healthz/ready\": read tcp 10.134.0.2:53502->10.134.0.218:15021: read: connection reset by peer"
Nov 03 01:03:18.870619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:18.870589    6000 generic.go:356] "Generic (PLEG): container finished" podID="06270d58-ffc7-4d70-afac-7f8f99570583" containerID="30cc1f9cc0cbf0dca64e56b307584c7db06fc0fc5a1e1d38c19029e24d90c91d" exitCode=0
Nov 03 01:03:18.870804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:18.870660    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" event={"ID":"06270d58-ffc7-4d70-afac-7f8f99570583","Type":"ContainerDied","Data":"30cc1f9cc0cbf0dca64e56b307584c7db06fc0fc5a1e1d38c19029e24d90c91d"}
Nov 03 01:03:19.022551 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.022524    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 01:03:19.062165 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062137    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-data\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062184    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-istio-token\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-istio-token\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062236    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-kube-api-access-brctk\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-kube-api-access-brctk\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062274    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-socket\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062323    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06270d58-ffc7-4d70-afac-7f8f99570583-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06270d58-ffc7-4d70-afac-7f8f99570583-istiod-ca-cert\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062393    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06270d58-ffc7-4d70-afac-7f8f99570583-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06270d58-ffc7-4d70-afac-7f8f99570583-istio-podinfo\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062420    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-certs\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062482    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-data" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:19.062615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062575    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-credential-socket\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062615    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-socket" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:19.062837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062670    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-envoy\") pod \"06270d58-ffc7-4d70-afac-7f8f99570583\" (UID: \"06270d58-ffc7-4d70-afac-7f8f99570583\") "
Nov 03 01:03:19.062837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062694    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-certs" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:19.062990 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062856    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-credential-socket" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:19.062990 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.062887    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06270d58-ffc7-4d70-afac-7f8f99570583-istiod-ca-cert" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:19.063404 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.063386    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.063548 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.063532    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.063548 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.063553    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.063664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.063568    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.063664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.063580    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06270d58-ffc7-4d70-afac-7f8f99570583-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.064921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.064895    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06270d58-ffc7-4d70-afac-7f8f99570583-istio-podinfo" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:03:19.065022 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.064917    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-envoy" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:19.065200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.065181    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-istio-token" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:19.065279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.065210    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-kube-api-access-brctk" pod "06270d58-ffc7-4d70-afac-7f8f99570583" (UID: "06270d58-ffc7-4d70-afac-7f8f99570583"). InnerVolumeSpecName "kube-api-access-brctk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:19.165193 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.165159    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06270d58-ffc7-4d70-afac-7f8f99570583-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.165193 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.165194    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.165428 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.165210    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brctk\" (UniqueName: \"kubernetes.io/projected/06270d58-ffc7-4d70-afac-7f8f99570583-kube-api-access-brctk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.165428 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.165224    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06270d58-ffc7-4d70-afac-7f8f99570583-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:19.883720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.883686    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"
Nov 03 01:03:19.883720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.883694    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27" event={"ID":"06270d58-ffc7-4d70-afac-7f8f99570583","Type":"ContainerDied","Data":"28aadc11698a58fb333bd4352222db5fabfe24f2a2ffec6500dcfd25e2dd4c71"}
Nov 03 01:03:19.884398 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.883761    6000 scope.go:122] "RemoveContainer" containerID="30cc1f9cc0cbf0dca64e56b307584c7db06fc0fc5a1e1d38c19029e24d90c91d"
Nov 03 01:03:19.921945 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:03:19.921912    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:03:19.948892 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.948815    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"]
Nov 03 01:03:19.960787 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:19.960755    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-850433-openshift-default-7584745dd9-kld27"]
Nov 03 01:03:20.637007 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:20.620937    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" path="/var/lib/kubelet/pods/06270d58-ffc7-4d70-afac-7f8f99570583/volumes"
Nov 03 01:03:21.357281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:21.353316    6000 patch_prober.go:29] interesting pod/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:03:21.357281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:21.353421    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:03:22.013342 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.013232    6000 generic.go:356] "Generic (PLEG): container finished" podID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerID="ab91dd6abce2634b2b283838312f4a316177327ae531ae4ce3fbd67f7da84486" exitCode=0
Nov 03 01:03:22.013676 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.013652    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" event={"ID":"356bea3c-a365-4f7a-81f5-35c9bd90b427","Type":"ContainerDied","Data":"ab91dd6abce2634b2b283838312f4a316177327ae531ae4ce3fbd67f7da84486"}
Nov 03 01:03:22.650473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.650444    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 01:03:22.773391 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773350    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-token\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-token\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.773595 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773528    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-credential-socket\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.773696 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773678    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-kube-api-access-dg889\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-kube-api-access-dg889\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.773781 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773765    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-data\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.773844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773822    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-socket\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.773912 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773872    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-credential-socket" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:22.773912 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773904    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/356bea3c-a365-4f7a-81f5-35c9bd90b427-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356bea3c-a365-4f7a-81f5-35c9bd90b427-istiod-ca-cert\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.774025 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773948    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-podinfo\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.774025 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.773991    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-envoy\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.774142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.774019    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-certs\") pod \"356bea3c-a365-4f7a-81f5-35c9bd90b427\" (UID: \"356bea3c-a365-4f7a-81f5-35c9bd90b427\") "
Nov 03 01:03:22.774142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.774093    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-data" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:22.774467 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.774301    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-socket" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:22.774845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.774815    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356bea3c-a365-4f7a-81f5-35c9bd90b427-istiod-ca-cert" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:03:22.774973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.774898    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.774973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.774921    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.774973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.774934    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.775404 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.775379    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-certs" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:22.777941 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.776881    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-token" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:22.777941 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.777097    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-podinfo" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:03:22.778294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.778271    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-kube-api-access-dg889" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "kube-api-access-dg889". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:03:22.778613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.778597    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-envoy" pod "356bea3c-a365-4f7a-81f5-35c9bd90b427" (UID: "356bea3c-a365-4f7a-81f5-35c9bd90b427"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:03:22.877731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.877652    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg889\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-kube-api-access-dg889\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.877731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.877680    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/356bea3c-a365-4f7a-81f5-35c9bd90b427-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.877731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.877691    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.877731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.877699    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.877731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.877707    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/356bea3c-a365-4f7a-81f5-35c9bd90b427-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:22.877731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:22.877716    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/356bea3c-a365-4f7a-81f5-35c9bd90b427-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:03:23.032675 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:23.032633    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9" event={"ID":"356bea3c-a365-4f7a-81f5-35c9bd90b427","Type":"ContainerDied","Data":"2d61eb2d656da825a36fe519699fe320ad8e65a37ac407e03d1442025b0263a4"}
Nov 03 01:03:23.032894 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:23.032769    6000 scope.go:122] "RemoveContainer" containerID="ab91dd6abce2634b2b283838312f4a316177327ae531ae4ce3fbd67f7da84486"
Nov 03 01:03:23.033019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:23.032999    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"
Nov 03 01:03:23.156018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:23.155920    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"]
Nov 03 01:03:23.183393 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:23.183355    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-ac338d-openshift-default-d94664777-75xd9"]
Nov 03 01:03:24.570077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:03:24.569999    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" path="/var/lib/kubelet/pods/356bea3c-a365-4f7a-81f5-35c9bd90b427/volumes"
Nov 03 01:03:49.899546 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:03:49.899507    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:04:18.621278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:18.621233    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"]
Nov 03 01:04:19.897872 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:04:19.897832    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:04:33.183176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:33.183140    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:04:33.650064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:33.650022    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk" podUID="997abdcd-62e3-489c-9150-653c4573d8e3" containerName="main" containerID="cri-o://d82674a4de0c0c819e37fab9a219dc1cfb56bbdb9a3a780d7ea6d256ac08996f" gracePeriod=45
Nov 03 01:04:33.811357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:33.811326    6000 generic.go:356] "Generic (PLEG): container finished" podID="997abdcd-62e3-489c-9150-653c4573d8e3" containerID="d82674a4de0c0c819e37fab9a219dc1cfb56bbdb9a3a780d7ea6d256ac08996f" exitCode=0
Nov 03 01:04:33.811545 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:33.811401    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk" event={"ID":"997abdcd-62e3-489c-9150-653c4573d8e3","Type":"ContainerDied","Data":"d82674a4de0c0c819e37fab9a219dc1cfb56bbdb9a3a780d7ea6d256ac08996f"}
Nov 03 01:04:34.092900 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.092470    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 01:04:34.208354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.207076    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-tmp-dir\") pod \"997abdcd-62e3-489c-9150-653c4573d8e3\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") "
Nov 03 01:04:34.208354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.207176    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/997abdcd-62e3-489c-9150-653c4573d8e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/997abdcd-62e3-489c-9150-653c4573d8e3-tls-certs\") pod \"997abdcd-62e3-489c-9150-653c4573d8e3\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") "
Nov 03 01:04:34.208354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.207241    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-dshm\") pod \"997abdcd-62e3-489c-9150-653c4573d8e3\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") "
Nov 03 01:04:34.208354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.207329    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/997abdcd-62e3-489c-9150-653c4573d8e3-kube-api-access-xxlkj\" (UniqueName: \"kubernetes.io/projected/997abdcd-62e3-489c-9150-653c4573d8e3-kube-api-access-xxlkj\") pod \"997abdcd-62e3-489c-9150-653c4573d8e3\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") "
Nov 03 01:04:34.208354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.207362    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-home\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-home\") pod \"997abdcd-62e3-489c-9150-653c4573d8e3\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") "
Nov 03 01:04:34.208354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.207521    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-model-cache\") pod \"997abdcd-62e3-489c-9150-653c4573d8e3\" (UID: \"997abdcd-62e3-489c-9150-653c4573d8e3\") "
Nov 03 01:04:34.209586 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.209415    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-model-cache" pod "997abdcd-62e3-489c-9150-653c4573d8e3" (UID: "997abdcd-62e3-489c-9150-653c4573d8e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:34.210052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.209743    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-tmp-dir" pod "997abdcd-62e3-489c-9150-653c4573d8e3" (UID: "997abdcd-62e3-489c-9150-653c4573d8e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:34.216620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.216540    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-home" pod "997abdcd-62e3-489c-9150-653c4573d8e3" (UID: "997abdcd-62e3-489c-9150-653c4573d8e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:34.282154 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.257506    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/997abdcd-62e3-489c-9150-653c4573d8e3-tls-certs" pod "997abdcd-62e3-489c-9150-653c4573d8e3" (UID: "997abdcd-62e3-489c-9150-653c4573d8e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:04:34.285288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.284503    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/997abdcd-62e3-489c-9150-653c4573d8e3-kube-api-access-xxlkj" pod "997abdcd-62e3-489c-9150-653c4573d8e3" (UID: "997abdcd-62e3-489c-9150-653c4573d8e3"). InnerVolumeSpecName "kube-api-access-xxlkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:04:34.296789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.296754    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-dshm" pod "997abdcd-62e3-489c-9150-653c4573d8e3" (UID: "997abdcd-62e3-489c-9150-653c4573d8e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:04:34.317155 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.317119    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:04:34.317441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.317422    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/997abdcd-62e3-489c-9150-653c4573d8e3-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:04:34.317542 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.317531    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:04:34.338946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.338925    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxlkj\" (UniqueName: \"kubernetes.io/projected/997abdcd-62e3-489c-9150-653c4573d8e3-kube-api-access-xxlkj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:04:34.339146 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.339133    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:04:34.339233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.339222    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/997abdcd-62e3-489c-9150-653c4573d8e3-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:04:34.820839 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.820811    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"
Nov 03 01:04:34.821018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.820853    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk" event={"ID":"997abdcd-62e3-489c-9150-653c4573d8e3","Type":"ContainerDied","Data":"92627f28949dae19dfe273ce639d4c86be2979d8704f0f8c14acf3c43620ffff"}
Nov 03 01:04:34.821018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.820906    6000 scope.go:122] "RemoveContainer" containerID="d82674a4de0c0c819e37fab9a219dc1cfb56bbdb9a3a780d7ea6d256ac08996f"
Nov 03 01:04:34.872345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.872316    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"]
Nov 03 01:04:34.883075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:34.883050    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-056a96/test-model-056a96-kserve-75fd754d5-vmxlk"]
Nov 03 01:04:36.571418 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:04:36.571386    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="997abdcd-62e3-489c-9150-653c4573d8e3" path="/var/lib/kubelet/pods/997abdcd-62e3-489c-9150-653c4573d8e3/volumes"
Nov 03 01:04:49.898328 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:04:49.898288    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:05:01.250623 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:01.250585    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"]
Nov 03 01:05:01.281318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:01.281277    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"]
Nov 03 01:05:06.251688 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.251636    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d" podUID="3fc532fe-49e8-404d-ab30-fd29bae9f031" containerName="payload-processing" containerID="cri-o://3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4" gracePeriod=25
Nov 03 01:05:06.282603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.282546    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" podUID="e049c769-a863-4f07-8100-3d5d62c681d3" containerName="payload-pre-processing" containerID="cri-o://5b347d3bf78d176b532b4c8f11e6cedfabc9d55a23e18f810fae7e7fec98bb8f" gracePeriod=25
Nov 03 01:05:06.561502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.561469    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 01:05:06.619483 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.619446    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fc532fe-49e8-404d-ab30-fd29bae9f031-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc532fe-49e8-404d-ab30-fd29bae9f031-plugins-config-volume\") pod \"3fc532fe-49e8-404d-ab30-fd29bae9f031\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") "
Nov 03 01:05:06.619799 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.619777    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fc532fe-49e8-404d-ab30-fd29bae9f031-plugins-config-volume" pod "3fc532fe-49e8-404d-ab30-fd29bae9f031" (UID: "3fc532fe-49e8-404d-ab30-fd29bae9f031"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:06.619890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.619815    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fc532fe-49e8-404d-ab30-fd29bae9f031-kube-api-access-5lrxg\" (UniqueName: \"kubernetes.io/projected/3fc532fe-49e8-404d-ab30-fd29bae9f031-kube-api-access-5lrxg\") pod \"3fc532fe-49e8-404d-ab30-fd29bae9f031\" (UID: \"3fc532fe-49e8-404d-ab30-fd29bae9f031\") "
Nov 03 01:05:06.620680 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.620650    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3fc532fe-49e8-404d-ab30-fd29bae9f031-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:06.622195 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.622171    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fc532fe-49e8-404d-ab30-fd29bae9f031-kube-api-access-5lrxg" pod "3fc532fe-49e8-404d-ab30-fd29bae9f031" (UID: "3fc532fe-49e8-404d-ab30-fd29bae9f031"). InnerVolumeSpecName "kube-api-access-5lrxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:06.722215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:06.722182    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrxg\" (UniqueName: \"kubernetes.io/projected/3fc532fe-49e8-404d-ab30-fd29bae9f031-kube-api-access-5lrxg\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:07.139181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.139151    6000 generic.go:356] "Generic (PLEG): container finished" podID="3fc532fe-49e8-404d-ab30-fd29bae9f031" containerID="3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4" exitCode=0
Nov 03 01:05:07.139426 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.139211    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"
Nov 03 01:05:07.139426 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.139236    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d" event={"ID":"3fc532fe-49e8-404d-ab30-fd29bae9f031","Type":"ContainerDied","Data":"3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4"}
Nov 03 01:05:07.139426 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.139315    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d" event={"ID":"3fc532fe-49e8-404d-ab30-fd29bae9f031","Type":"ContainerDied","Data":"a56762cb16f733882dadc0afdd667ffc73e6a97734fa60c3b840e7e0a65570f7"}
Nov 03 01:05:07.139426 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.139347    6000 scope.go:122] "RemoveContainer" containerID="3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4"
Nov 03 01:05:07.152363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.152334    6000 scope.go:122] "RemoveContainer" containerID="3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4"
Nov 03 01:05:07.152706 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:05:07.152687    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4\": container with ID starting with 3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4 not found: ID does not exist" containerID="3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4"
Nov 03 01:05:07.152777 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.152714    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4"} err="failed to get container status \"3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4\": rpc error: code = NotFound desc = could not find container \"3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4\": container with ID starting with 3744b4cc4b4e6a4e4743bd6ef54fa199c9d618d2f50123761c64c724d3df03b4 not found: ID does not exist"
Nov 03 01:05:07.189747 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.189715    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"]
Nov 03 01:05:07.204509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.204479    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-49d7e5-99dbd976f-6vr6d"]
Nov 03 01:05:07.722717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:07.722679    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" podUID="e049c769-a863-4f07-8100-3d5d62c681d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:05:08.569632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:08.569604    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fc532fe-49e8-404d-ab30-fd29bae9f031" path="/var/lib/kubelet/pods/3fc532fe-49e8-404d-ab30-fd29bae9f031/volumes"
Nov 03 01:05:17.723295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:17.723236    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" podUID="e049c769-a863-4f07-8100-3d5d62c681d3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:05:19.897768 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:05:19.897728    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:05:20.409110 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:20.409074    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"]
Nov 03 01:05:20.440532 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:20.440501    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"]
Nov 03 01:05:25.410348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.410301    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg" podUID="26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" containerName="payload-processing" containerID="cri-o://8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae" gracePeriod=25
Nov 03 01:05:25.441475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.441437    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" podUID="f7529e5d-44c6-4282-b229-a977f6377256" containerName="payload-pre-processing" containerID="cri-o://9169802c62d48b5f6667e6b69b6f025c2133eaffae93f4da676c45305cea3c4d" gracePeriod=25
Nov 03 01:05:25.710178 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.710150    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 01:05:25.728194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.728162    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-kube-api-access-5m97v\" (UniqueName: \"kubernetes.io/projected/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-kube-api-access-5m97v\") pod \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") "
Nov 03 01:05:25.728396 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.728341    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-plugins-config-volume\") pod \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\" (UID: \"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1\") "
Nov 03 01:05:25.728766 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.728741    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-plugins-config-volume" pod "26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" (UID: "26767d83-e7c7-444c-bc3c-b9e4a71bc8d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:25.729443 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.729417    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:25.731109 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.730893    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-kube-api-access-5m97v" pod "26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" (UID: "26767d83-e7c7-444c-bc3c-b9e4a71bc8d1"). InnerVolumeSpecName "kube-api-access-5m97v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:25.831317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:25.831284    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5m97v\" (UniqueName: \"kubernetes.io/projected/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1-kube-api-access-5m97v\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:26.452455 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.452414    6000 generic.go:356] "Generic (PLEG): container finished" podID="e049c769-a863-4f07-8100-3d5d62c681d3" containerID="5b347d3bf78d176b532b4c8f11e6cedfabc9d55a23e18f810fae7e7fec98bb8f" exitCode=0
Nov 03 01:05:26.452910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.452504    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" event={"ID":"e049c769-a863-4f07-8100-3d5d62c681d3","Type":"ContainerDied","Data":"5b347d3bf78d176b532b4c8f11e6cedfabc9d55a23e18f810fae7e7fec98bb8f"}
Nov 03 01:05:26.454461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.454433    6000 generic.go:356] "Generic (PLEG): container finished" podID="26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" containerID="8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae" exitCode=0
Nov 03 01:05:26.454625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.454492    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"
Nov 03 01:05:26.454625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.454497    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg" event={"ID":"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1","Type":"ContainerDied","Data":"8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae"}
Nov 03 01:05:26.454625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.454550    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg" event={"ID":"26767d83-e7c7-444c-bc3c-b9e4a71bc8d1","Type":"ContainerDied","Data":"6a2e3cff76abae41eeb5478f16dab321826b62aaa82b8b9892732aa7e1401af4"}
Nov 03 01:05:26.454625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.454580    6000 scope.go:122] "RemoveContainer" containerID="8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae"
Nov 03 01:05:26.479401 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.479375    6000 scope.go:122] "RemoveContainer" containerID="8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae"
Nov 03 01:05:26.479734 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:05:26.479712    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae\": container with ID starting with 8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae not found: ID does not exist" containerID="8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae"
Nov 03 01:05:26.479830 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.479754    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae"} err="failed to get container status \"8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae\": rpc error: code = NotFound desc = could not find container \"8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae\": container with ID starting with 8f43bfd5455b8b90bc429436b7b7b32633ca47022ddf35709b113e4f3a0f7cae not found: ID does not exist"
Nov 03 01:05:26.511977 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.511947    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"]
Nov 03 01:05:26.519841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.519816    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-7f20f7-d7cc67c75-vdjtg"]
Nov 03 01:05:26.544528 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.544501    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 01:05:26.572683 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.572609    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" path="/var/lib/kubelet/pods/26767d83-e7c7-444c-bc3c-b9e4a71bc8d1/volumes"
Nov 03 01:05:26.644716 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.644686    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e049c769-a863-4f07-8100-3d5d62c681d3-kube-api-access-vjpfj\" (UniqueName: \"kubernetes.io/projected/e049c769-a863-4f07-8100-3d5d62c681d3-kube-api-access-vjpfj\") pod \"e049c769-a863-4f07-8100-3d5d62c681d3\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") "
Nov 03 01:05:26.644927 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.644757    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e049c769-a863-4f07-8100-3d5d62c681d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049c769-a863-4f07-8100-3d5d62c681d3-plugins-config-volume\") pod \"e049c769-a863-4f07-8100-3d5d62c681d3\" (UID: \"e049c769-a863-4f07-8100-3d5d62c681d3\") "
Nov 03 01:05:26.645218 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.645187    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e049c769-a863-4f07-8100-3d5d62c681d3-plugins-config-volume" pod "e049c769-a863-4f07-8100-3d5d62c681d3" (UID: "e049c769-a863-4f07-8100-3d5d62c681d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:26.645770 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.645744    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e049c769-a863-4f07-8100-3d5d62c681d3-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:26.647152 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.647128    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e049c769-a863-4f07-8100-3d5d62c681d3-kube-api-access-vjpfj" pod "e049c769-a863-4f07-8100-3d5d62c681d3" (UID: "e049c769-a863-4f07-8100-3d5d62c681d3"). InnerVolumeSpecName "kube-api-access-vjpfj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:26.747113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:26.747083    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjpfj\" (UniqueName: \"kubernetes.io/projected/e049c769-a863-4f07-8100-3d5d62c681d3-kube-api-access-vjpfj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:27.463679 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:27.463639    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"
Nov 03 01:05:27.464200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:27.463632    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2" event={"ID":"e049c769-a863-4f07-8100-3d5d62c681d3","Type":"ContainerDied","Data":"9c430a27276e94e96c56a730efc69c7e73e87d4c4e59df6b8048dfab9741cb42"}
Nov 03 01:05:27.464200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:27.463835    6000 scope.go:122] "RemoveContainer" containerID="5b347d3bf78d176b532b4c8f11e6cedfabc9d55a23e18f810fae7e7fec98bb8f"
Nov 03 01:05:27.515979 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:27.515947    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"]
Nov 03 01:05:27.525381 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:27.525352    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-49d7e5-56895dc9f9-vwjz2"]
Nov 03 01:05:28.568868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:28.568835    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e049c769-a863-4f07-8100-3d5d62c681d3" path="/var/lib/kubelet/pods/e049c769-a863-4f07-8100-3d5d62c681d3/volumes"
Nov 03 01:05:29.179897 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:29.179854    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" podUID="f7529e5d-44c6-4282-b229-a977f6377256" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:05:31.921179 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:31.921135    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"]
Nov 03 01:05:31.922751 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:31.922714    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" containerID="cri-o://1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef" gracePeriod=30
Nov 03 01:05:33.662567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:33.662533    6000 patch_prober.go:29] interesting pod/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:05:33.663056 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:33.662595    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:05:37.222239 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.222216    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 01:05:37.341589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341553    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-token\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-token\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.341802 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341700    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-podinfo\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.341802 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341736    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istiod-ca-cert\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.341802 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341764    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-certs\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.341976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341799    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-credential-socket\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.341976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341834    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-envoy\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.341976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341877    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-data\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.342142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.341995    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-socket\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.342197 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.342127    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-credential-socket" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:37.342197 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.342146    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-kube-api-access-vs45t\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-kube-api-access-vs45t\") pod \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\" (UID: \"98fda859-71bf-4b39-bb5c-5a26bd7ef43a\") "
Nov 03 01:05:37.342339 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.342212    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-data" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:37.342339 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.342292    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istiod-ca-cert" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:37.342476 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.342444    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-certs" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:37.342633 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.342611    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-socket" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:37.343199 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.343126    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.343199 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.343155    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.343199 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.343169    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.343199 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.343182    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.343199 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.343195    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.344327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.344301    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-token" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:37.344777 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.344757    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-kube-api-access-vs45t" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "kube-api-access-vs45t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:37.344877 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.344851    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-podinfo" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:05:37.345349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.345331    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-envoy" pod "98fda859-71bf-4b39-bb5c-5a26bd7ef43a" (UID: "98fda859-71bf-4b39-bb5c-5a26bd7ef43a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:05:37.396281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.396237    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:05:37.444438 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.444413    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.444438 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.444437    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs45t\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-kube-api-access-vs45t\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.444922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.444446    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.444922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.444455    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98fda859-71bf-4b39-bb5c-5a26bd7ef43a-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:37.547031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.546942    6000 generic.go:356] "Generic (PLEG): container finished" podID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerID="1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef" exitCode=0
Nov 03 01:05:37.547031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.547003    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"
Nov 03 01:05:37.547031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.547013    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" event={"ID":"98fda859-71bf-4b39-bb5c-5a26bd7ef43a","Type":"ContainerDied","Data":"1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef"}
Nov 03 01:05:37.547366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.547084    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn" event={"ID":"98fda859-71bf-4b39-bb5c-5a26bd7ef43a","Type":"ContainerDied","Data":"20c73db07e103ec352fc29835afe9a7623faa72bcdd55b2319b519246dec4583"}
Nov 03 01:05:37.547366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.547105    6000 scope.go:122] "RemoveContainer" containerID="1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef"
Nov 03 01:05:37.559847 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.559827    6000 scope.go:122] "RemoveContainer" containerID="3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7"
Nov 03 01:05:37.571697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.571680    6000 scope.go:122] "RemoveContainer" containerID="1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef"
Nov 03 01:05:37.571998 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:05:37.571978    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef\": container with ID starting with 1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef not found: ID does not exist" containerID="1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef"
Nov 03 01:05:37.572088 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.572011    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef"} err="failed to get container status \"1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef\": rpc error: code = NotFound desc = could not find container \"1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef\": container with ID starting with 1ed2bc2d09fdad7d80c4d52ccccff204a71f2ca05dd4f451e21ff824e25928ef not found: ID does not exist"
Nov 03 01:05:37.572088 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.572035    6000 scope.go:122] "RemoveContainer" containerID="3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7"
Nov 03 01:05:37.572374 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:05:37.572351    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7\": container with ID starting with 3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7 not found: ID does not exist" containerID="3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7"
Nov 03 01:05:37.572425 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.572381    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7"} err="failed to get container status \"3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7\": rpc error: code = NotFound desc = could not find container \"3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7\": container with ID starting with 3527ffc19bd3a0ff9a1af932f914918ea684367a7a5e6d3a608a61da1a5d43e7 not found: ID does not exist"
Nov 03 01:05:37.598064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.598029    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"]
Nov 03 01:05:37.604462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:37.604438    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-883a9e-openshift-default-567bfd88dc-gvwpn"]
Nov 03 01:05:38.539269 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:38.539219    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"]
Nov 03 01:05:38.570971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:38.570936    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" path="/var/lib/kubelet/pods/98fda859-71bf-4b39-bb5c-5a26bd7ef43a/volumes"
Nov 03 01:05:39.227407 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:39.227365    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" podUID="f7529e5d-44c6-4282-b229-a977f6377256" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:05:43.539710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:43.539657    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" podUID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" containerName="payload-processing" containerID="cri-o://7086da36cdd72eb4668151c741570d942b43e7960cbbeb8d3bfa771df9a805e6" gracePeriod=25
Nov 03 01:05:45.674804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.674771    6000 generic.go:356] "Generic (PLEG): container finished" podID="f7529e5d-44c6-4282-b229-a977f6377256" containerID="9169802c62d48b5f6667e6b69b6f025c2133eaffae93f4da676c45305cea3c4d" exitCode=0
Nov 03 01:05:45.675365 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.674858    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" event={"ID":"f7529e5d-44c6-4282-b229-a977f6377256","Type":"ContainerDied","Data":"9169802c62d48b5f6667e6b69b6f025c2133eaffae93f4da676c45305cea3c4d"}
Nov 03 01:05:45.731215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.731183    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 01:05:45.805976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.805944    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7529e5d-44c6-4282-b229-a977f6377256-kube-api-access-b8whs\" (UniqueName: \"kubernetes.io/projected/f7529e5d-44c6-4282-b229-a977f6377256-kube-api-access-b8whs\") pod \"f7529e5d-44c6-4282-b229-a977f6377256\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") "
Nov 03 01:05:45.806173 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.806074    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7529e5d-44c6-4282-b229-a977f6377256-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7529e5d-44c6-4282-b229-a977f6377256-plugins-config-volume\") pod \"f7529e5d-44c6-4282-b229-a977f6377256\" (UID: \"f7529e5d-44c6-4282-b229-a977f6377256\") "
Nov 03 01:05:45.806495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.806470    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7529e5d-44c6-4282-b229-a977f6377256-plugins-config-volume" pod "f7529e5d-44c6-4282-b229-a977f6377256" (UID: "f7529e5d-44c6-4282-b229-a977f6377256"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:05:45.806988 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.806966    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7529e5d-44c6-4282-b229-a977f6377256-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:45.808439 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.808407    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7529e5d-44c6-4282-b229-a977f6377256-kube-api-access-b8whs" pod "f7529e5d-44c6-4282-b229-a977f6377256" (UID: "f7529e5d-44c6-4282-b229-a977f6377256"). InnerVolumeSpecName "kube-api-access-b8whs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:05:45.908760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:45.908729    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8whs\" (UniqueName: \"kubernetes.io/projected/f7529e5d-44c6-4282-b229-a977f6377256-kube-api-access-b8whs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:05:46.683850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:46.683818    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb" event={"ID":"f7529e5d-44c6-4282-b229-a977f6377256","Type":"ContainerDied","Data":"cd90c0411bea48e65bf942cb64f02c40706e595ec5b497ba611f008c63cf2436"}
Nov 03 01:05:46.683850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:46.683840    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"
Nov 03 01:05:46.684381 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:46.683875    6000 scope.go:122] "RemoveContainer" containerID="9169802c62d48b5f6667e6b69b6f025c2133eaffae93f4da676c45305cea3c4d"
Nov 03 01:05:46.733207 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:46.733178    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"]
Nov 03 01:05:46.746132 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:46.746107    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-7f20f7-795ff8bb47-6xxnb"]
Nov 03 01:05:48.550922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:48.550874    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" podUID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:05:48.569364 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:48.569335    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7529e5d-44c6-4282-b229-a977f6377256" path="/var/lib/kubelet/pods/f7529e5d-44c6-4282-b229-a977f6377256/volumes"
Nov 03 01:05:49.899318 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:05:49.899281    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:05:58.558700 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:05:58.558655    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" podUID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:06:03.830985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:03.830894    6000 generic.go:356] "Generic (PLEG): container finished" podID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" containerID="7086da36cdd72eb4668151c741570d942b43e7960cbbeb8d3bfa771df9a805e6" exitCode=0
Nov 03 01:06:03.831534 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:03.831029    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" event={"ID":"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4","Type":"ContainerDied","Data":"7086da36cdd72eb4668151c741570d942b43e7960cbbeb8d3bfa771df9a805e6"}
Nov 03 01:06:03.881235 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:03.881204    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 01:06:04.029182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.029145    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-plugins-config-volume\") pod \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") "
Nov 03 01:06:04.029447 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.029433    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-kube-api-access-fwjbb\" (UniqueName: \"kubernetes.io/projected/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-kube-api-access-fwjbb\") pod \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\" (UID: \"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4\") "
Nov 03 01:06:04.029572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.029546    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-plugins-config-volume" pod "7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" (UID: "7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:04.029903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.029887    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:04.031707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.031684    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-kube-api-access-fwjbb" pod "7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" (UID: "7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4"). InnerVolumeSpecName "kube-api-access-fwjbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:04.131792 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.131756    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fwjbb\" (UniqueName: \"kubernetes.io/projected/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4-kube-api-access-fwjbb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:04.840146 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.840063    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"
Nov 03 01:06:04.840146 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.840104    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6" event={"ID":"7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4","Type":"ContainerDied","Data":"563431d7c9381c0ef633966ed8da6c3d22c281c75e4212f493637db231764465"}
Nov 03 01:06:04.840715 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.840165    6000 scope.go:122] "RemoveContainer" containerID="7086da36cdd72eb4668151c741570d942b43e7960cbbeb8d3bfa771df9a805e6"
Nov 03 01:06:04.890609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.890581    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"]
Nov 03 01:06:04.902805 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:04.902770    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-056a96-6d9bf47485-58kz6"]
Nov 03 01:06:06.570405 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:06.570374    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" path="/var/lib/kubelet/pods/7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4/volumes"
Nov 03 01:06:19.897482 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:06:19.897402    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:06:20.820565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:20.820529    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"]
Nov 03 01:06:20.820766 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:20.820746    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk" podUID="08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" containerName="maas-api" containerID="cri-o://537e43e9eb85502508dda77fc152dbdd89be351df2d075ebb7da5de1c6485011" gracePeriod=30
Nov 03 01:06:21.000761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.000728    6000 generic.go:356] "Generic (PLEG): container finished" podID="08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" containerID="537e43e9eb85502508dda77fc152dbdd89be351df2d075ebb7da5de1c6485011" exitCode=0
Nov 03 01:06:21.000761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.000765    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk" event={"ID":"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f","Type":"ContainerDied","Data":"537e43e9eb85502508dda77fc152dbdd89be351df2d075ebb7da5de1c6485011"}
Nov 03 01:06:21.235577 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.235535    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 01:06:21.415501 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.414032    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-kube-api-access-6x7t4\" (UniqueName: \"kubernetes.io/projected/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-kube-api-access-6x7t4\") pod \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") "
Nov 03 01:06:21.415501 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.414082    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-maas-api-tls\") pod \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") "
Nov 03 01:06:21.415501 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.414435    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-metrics-certs\") pod \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") "
Nov 03 01:06:21.415501 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.414553    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-openshift-service-ca\") pod \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\" (UID: \"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f\") "
Nov 03 01:06:21.416245 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.416208    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-openshift-service-ca" pod "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" (UID: "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:21.416554 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.416521    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:21.421457 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.421193    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-maas-api-tls" pod "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" (UID: "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:21.421963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.421925    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-metrics-certs" pod "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" (UID: "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:21.426533 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.426509    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-kube-api-access-6x7t4" pod "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" (UID: "08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f"). InnerVolumeSpecName "kube-api-access-6x7t4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:21.517867 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.517836    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:21.517867 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.517865    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x7t4\" (UniqueName: \"kubernetes.io/projected/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-kube-api-access-6x7t4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:21.517867 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:21.517874    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:22.009457 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:22.009418    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"
Nov 03 01:06:22.009914 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:22.009464    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk" event={"ID":"08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f","Type":"ContainerDied","Data":"ae2b253d5568bd571766bdebd0f8e9a0d22c4c8aeda86ea4f7d328258d1fee95"}
Nov 03 01:06:22.009914 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:22.009534    6000 scope.go:122] "RemoveContainer" containerID="537e43e9eb85502508dda77fc152dbdd89be351df2d075ebb7da5de1c6485011"
Nov 03 01:06:22.059078 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:22.059051    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"]
Nov 03 01:06:22.064306 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:22.064281    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-056a96-7c9c7656dd-gr9rk"]
Nov 03 01:06:22.570465 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:22.570436    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" path="/var/lib/kubelet/pods/08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f/volumes"
Nov 03 01:06:27.844693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:27.844638    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"]
Nov 03 01:06:27.845569 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:27.845512    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy" containerID="cri-o://5c2a594a3a053e7a3ed6c75ce6aac973fbb8919f5ad771c520aff7c5c2997472" gracePeriod=30
Nov 03 01:06:33.139120 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.139019    6000 generic.go:356] "Generic (PLEG): container finished" podID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerID="5c2a594a3a053e7a3ed6c75ce6aac973fbb8919f5ad771c520aff7c5c2997472" exitCode=0
Nov 03 01:06:33.139120 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.139087    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" event={"ID":"b63488ad-6f80-48ed-8b9e-66ca54e9fff1","Type":"ContainerDied","Data":"5c2a594a3a053e7a3ed6c75ce6aac973fbb8919f5ad771c520aff7c5c2997472"}
Nov 03 01:06:33.540635 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.540605    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 01:06:33.609058 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609022    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-podinfo\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609069    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-kube-api-access-g9gz8\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-kube-api-access-g9gz8\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609105    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-data\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609191    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-credential-socket\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609282    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-envoy\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609315    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-socket\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609349    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-token\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-token\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609443    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istiod-ca-cert\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609511    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-certs\") pod \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\" (UID: \"b63488ad-6f80-48ed-8b9e-66ca54e9fff1\") "
Nov 03 01:06:33.609731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609610    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-socket" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.609934 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609885    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-data" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.609934 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.609928    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istiod-ca-cert" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:33.610209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.610140    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-certs" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.610565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.610431    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-credential-socket" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.610565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.610497    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.610565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.610519    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.610565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.610531    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.610565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.610543    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.612678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.612613    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-token" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:33.616505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.616482    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-kube-api-access-g9gz8" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "kube-api-access-g9gz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:33.616761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.616740    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-envoy" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:33.627915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.627870    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-podinfo" pod "b63488ad-6f80-48ed-8b9e-66ca54e9fff1" (UID: "b63488ad-6f80-48ed-8b9e-66ca54e9fff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:33.712475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.712381    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.712475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.712417    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9gz8\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-kube-api-access-g9gz8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.712475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.712431    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.712475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.712444    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:33.712475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:33.712455    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b63488ad-6f80-48ed-8b9e-66ca54e9fff1-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:34.147714 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:34.147684    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"
Nov 03 01:06:34.148169 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:34.147712    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5" event={"ID":"b63488ad-6f80-48ed-8b9e-66ca54e9fff1","Type":"ContainerDied","Data":"9707f4c4341bdd01e7b85f1f9bfe497142b475467d17a8a472b6f3232cc4e1b5"}
Nov 03 01:06:34.148169 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:34.147769    6000 scope.go:122] "RemoveContainer" containerID="5c2a594a3a053e7a3ed6c75ce6aac973fbb8919f5ad771c520aff7c5c2997472"
Nov 03 01:06:34.209345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:34.209315    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"]
Nov 03 01:06:34.213906 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:34.213882    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-056a96-openshift-default-777fc45c8c-jkhd5"]
Nov 03 01:06:34.569309 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:34.569207    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" path="/var/lib/kubelet/pods/b63488ad-6f80-48ed-8b9e-66ca54e9fff1/volumes"
Nov 03 01:06:35.364019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:35.363978    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"]
Nov 03 01:06:35.364588 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:35.364195    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" containerID="cri-o://a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5" gracePeriod=30
Nov 03 01:06:38.117428 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:38.117387    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"]
Nov 03 01:06:40.194330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.194289    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:06:40.194767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.194352    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:06:40.662174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.662146    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 01:06:40.747743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.747653    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-data\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.747743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.747722    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-socket\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.747960 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.747824    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-credential-socket\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.747960 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.747943    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-envoy\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.748048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748011    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-socket" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:40.748048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748006    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-data" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:40.748127 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748089    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-certs\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.748168 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748136    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istiod-ca-cert\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.748210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748173    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-kube-api-access-kcl5d\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-kube-api-access-kcl5d\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.748248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748212    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-podinfo\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.748248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748220    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-credential-socket" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:40.748355 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748322    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-token\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-token\") pod \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\" (UID: \"1273c82d-11d9-4a6a-aff7-a190d9e419d5\") "
Nov 03 01:06:40.748640 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748616    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-certs" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:40.748845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.748640    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istiod-ca-cert" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:40.749111 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.749089    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.749215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.749119    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.749215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.749162    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.749397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.749328    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.749397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.749344    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.750727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.750702    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-envoy" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:40.751142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.751119    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-kube-api-access-kcl5d" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "kube-api-access-kcl5d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:40.751565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.751541    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-podinfo" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:40.752204 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.752173    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-token" pod "1273c82d-11d9-4a6a-aff7-a190d9e419d5" (UID: "1273c82d-11d9-4a6a-aff7-a190d9e419d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:40.850617 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.850585    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcl5d\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-kube-api-access-kcl5d\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.850801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.850627    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.850801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.850637    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:40.850801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:40.850645    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1273c82d-11d9-4a6a-aff7-a190d9e419d5-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:41.205535 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.205503    6000 generic.go:356] "Generic (PLEG): container finished" podID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerID="a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5" exitCode=0
Nov 03 01:06:41.205959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.205551    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"
Nov 03 01:06:41.205959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.205583    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" event={"ID":"1273c82d-11d9-4a6a-aff7-a190d9e419d5","Type":"ContainerDied","Data":"a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5"}
Nov 03 01:06:41.205959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.205640    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd" event={"ID":"1273c82d-11d9-4a6a-aff7-a190d9e419d5","Type":"ContainerDied","Data":"dd763c06aa184e60d5eda68a9fbd332a81f78c58061e3493322781335fa92a82"}
Nov 03 01:06:41.205959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.205670    6000 scope.go:122] "RemoveContainer" containerID="a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5"
Nov 03 01:06:41.220367 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.220351    6000 scope.go:122] "RemoveContainer" containerID="a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5"
Nov 03 01:06:41.220597 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:06:41.220582    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5\": container with ID starting with a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5 not found: ID does not exist" containerID="a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5"
Nov 03 01:06:41.220656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.220606    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5"} err="failed to get container status \"a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5\": rpc error: code = NotFound desc = could not find container \"a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5\": container with ID starting with a9be91452c74cdd16ac60a2dbdff787d24746b590e785ac2499257fa923e81f5 not found: ID does not exist"
Nov 03 01:06:41.255747 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.255716    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"]
Nov 03 01:06:41.267002 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.266975    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e62935-openshift-default-7684cbb99f-5k7jd"]
Nov 03 01:06:41.544470 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.544329    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"]
Nov 03 01:06:41.544679 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:41.544602    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" containerID="cri-o://7f46da08973d0c0b2e733f64d0bda577f692f7ec49ec23b460ecd5d681bc16c6" gracePeriod=30
Nov 03 01:06:42.571363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:42.571329    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" path="/var/lib/kubelet/pods/1273c82d-11d9-4a6a-aff7-a190d9e419d5/volumes"
Nov 03 01:06:43.522150 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:43.522118    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:06:43.522333 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:43.522172    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:06:44.253502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:44.253461    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:06:47.734215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.734178    6000 generic.go:356] "Generic (PLEG): container finished" podID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerID="7f46da08973d0c0b2e733f64d0bda577f692f7ec49ec23b460ecd5d681bc16c6" exitCode=0
Nov 03 01:06:47.735057 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.735029    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" event={"ID":"dda4f231-46f1-4753-bb41-d5e8745dd7d5","Type":"ContainerDied","Data":"7f46da08973d0c0b2e733f64d0bda577f692f7ec49ec23b460ecd5d681bc16c6"}
Nov 03 01:06:47.765297 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.737788    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 01:06:47.914282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.910766    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-data\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:47.914282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.910840    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-certs\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:47.914282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.910881    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-kube-api-access-4c8zr\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-kube-api-access-4c8zr\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:47.914282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.910968    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-envoy\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:47.914282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.911014    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-token\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-token\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:47.914282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.911093    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-credential-socket\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:47.914282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.911140    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istiod-ca-cert\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:47.947280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:47.941336    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-data" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:48.022743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.000367    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-credential-socket" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:48.022743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.000493    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 03 01:06:48.022743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.005299    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-certs" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:48.022743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.006687    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istiod-ca-cert" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:06:48.032286 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.027915    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-kube-api-access-4c8zr" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "kube-api-access-4c8zr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:48.032519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.031443    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-podinfo\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:48.032743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.032719    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-socket\") pod \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\" (UID: \"dda4f231-46f1-4753-bb41-d5e8745dd7d5\") "
Nov 03 01:06:48.068909 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.058427    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-socket" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:48.069205 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.069179    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.069383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.069367    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.073278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.069485    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4c8zr\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-kube-api-access-4c8zr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.073509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.073489    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.089277 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.073626    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.144045 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.144004    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-podinfo" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:06:48.202707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.202220    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.202707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.202274    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.202707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.202435    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-token" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:48.238446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.238405    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-envoy" pod "dda4f231-46f1-4753-bb41-d5e8745dd7d5" (UID: "dda4f231-46f1-4753-bb41-d5e8745dd7d5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:48.304593 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.304476    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.304593 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.304511    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dda4f231-46f1-4753-bb41-d5e8745dd7d5-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:48.326551 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.326512    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/ready\": context deadline exceeded"
Nov 03 01:06:48.338625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.338578    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" probeResult="failure" output="Get \"https://10.134.0.63:8000/ready\": context deadline exceeded"
Nov 03 01:06:48.501359 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.498362    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn" podUID="b703fed3-ecc3-44fc-9275-0831d71e9e1c" containerName="main" probeResult="failure" output="Get \"https://10.134.1.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.609280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.606576    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.609280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.608116    6000 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" probeResult="failure" output="Get \"https://10.134.0.73:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.846387 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.846338    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" probeResult="failure" output="Get \"https://10.134.0.70:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:48.973858 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.973764    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2" event={"ID":"dda4f231-46f1-4753-bb41-d5e8745dd7d5","Type":"ContainerDied","Data":"c15bba0c1b6049e5df6ea24af3f29ee6a561aa9bb5483dd8cb67280d4eb3eb26"}
Nov 03 01:06:48.974128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.974107    6000 scope.go:122] "RemoveContainer" containerID="7f46da08973d0c0b2e733f64d0bda577f692f7ec49ec23b460ecd5d681bc16c6"
Nov 03 01:06:48.974444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:48.974424    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"
Nov 03 01:06:49.164976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:49.163582    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"]
Nov 03 01:06:49.183644 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:49.183593    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-dr4b2"]
Nov 03 01:06:49.281011 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:49.280918    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w5m48 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:06:49.281350 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:49.281289    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:49.897992 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:06:49.897955    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:06:50.600135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:50.600100    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" path="/var/lib/kubelet/pods/dda4f231-46f1-4753-bb41-d5e8745dd7d5/volumes"
Nov 03 01:06:53.150246 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.150198    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9" podUID="6a83d623-1d5f-4dfa-8b5a-f826ec572c44" containerName="main" containerID="cri-o://26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526" gracePeriod=45
Nov 03 01:06:53.456979 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.456952    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:06:53.596082 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.596047    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tmp-dir\") pod \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") "
Nov 03 01:06:53.596427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.596403    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tls-certs\" (UniqueName: \"kubernetes.io/secret/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tls-certs\") pod \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") "
Nov 03 01:06:53.596964 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.596677    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tmp-dir" pod "6a83d623-1d5f-4dfa-8b5a-f826ec572c44" (UID: "6a83d623-1d5f-4dfa-8b5a-f826ec572c44"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.597178 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.597116    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-dshm\") pod \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") "
Nov 03 01:06:53.597369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.597349    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-model-cache\") pod \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") "
Nov 03 01:06:53.597641 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.597618    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-model-cache" pod "6a83d623-1d5f-4dfa-8b5a-f826ec572c44" (UID: "6a83d623-1d5f-4dfa-8b5a-f826ec572c44"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.597838 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.597820    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-kube-api-access-jg4fk\" (UniqueName: \"kubernetes.io/projected/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-kube-api-access-jg4fk\") pod \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") "
Nov 03 01:06:53.598498 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.598471    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-home\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-home\") pod \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\" (UID: \"6a83d623-1d5f-4dfa-8b5a-f826ec572c44\") "
Nov 03 01:06:53.599284 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.599164    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.599284 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.599189    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.599543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.599346    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-home" pod "6a83d623-1d5f-4dfa-8b5a-f826ec572c44" (UID: "6a83d623-1d5f-4dfa-8b5a-f826ec572c44"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.600513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.600491    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-dshm" pod "6a83d623-1d5f-4dfa-8b5a-f826ec572c44" (UID: "6a83d623-1d5f-4dfa-8b5a-f826ec572c44"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:06:53.600784 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.600768    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tls-certs" pod "6a83d623-1d5f-4dfa-8b5a-f826ec572c44" (UID: "6a83d623-1d5f-4dfa-8b5a-f826ec572c44"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:06:53.600934 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.600918    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-kube-api-access-jg4fk" pod "6a83d623-1d5f-4dfa-8b5a-f826ec572c44" (UID: "6a83d623-1d5f-4dfa-8b5a-f826ec572c44"). InnerVolumeSpecName "kube-api-access-jg4fk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:06:53.700855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.700785    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.700855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.700810    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.700855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.700820    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg4fk\" (UniqueName: \"kubernetes.io/projected/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-kube-api-access-jg4fk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:53.700855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:53.700828    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a83d623-1d5f-4dfa-8b5a-f826ec572c44-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:06:54.013395 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.013308    6000 generic.go:356] "Generic (PLEG): container finished" podID="6a83d623-1d5f-4dfa-8b5a-f826ec572c44" containerID="26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526" exitCode=0
Nov 03 01:06:54.013395 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.013357    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"
Nov 03 01:06:54.013618 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.013396    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9" event={"ID":"6a83d623-1d5f-4dfa-8b5a-f826ec572c44","Type":"ContainerDied","Data":"26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526"}
Nov 03 01:06:54.013618 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.013444    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9" event={"ID":"6a83d623-1d5f-4dfa-8b5a-f826ec572c44","Type":"ContainerDied","Data":"d3e18954cbea65f14326209bde758bc3a66b7cd095009099428f41a3f4152287"}
Nov 03 01:06:54.013618 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.013464    6000 scope.go:122] "RemoveContainer" containerID="26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526"
Nov 03 01:06:54.042105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.042028    6000 scope.go:122] "RemoveContainer" containerID="26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526"
Nov 03 01:06:54.042488 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:06:54.042457    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526\": container with ID starting with 26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526 not found: ID does not exist" containerID="26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526"
Nov 03 01:06:54.042600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.042498    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526"} err="failed to get container status \"26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526\": rpc error: code = NotFound desc = could not find container \"26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526\": container with ID starting with 26553306b8bd6785f1f54b43cc6e409886dee9784e4a203f19084f6523e34526 not found: ID does not exist"
Nov 03 01:06:54.067507 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.067473    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"]
Nov 03 01:06:54.071845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.071820    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9"]
Nov 03 01:06:54.442026 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.441983    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-8ed55a/test-model-8ed55a-kserve-86774654c5-r96l9" podUID="6a83d623-1d5f-4dfa-8b5a-f826ec572c44" containerName="main" probeResult="failure" output="Get \"https://10.134.1.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:06:54.568872 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:06:54.568836    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a83d623-1d5f-4dfa-8b5a-f826ec572c44" path="/var/lib/kubelet/pods/6a83d623-1d5f-4dfa-8b5a-f826ec572c44/volumes"
Nov 03 01:07:03.546826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:03.546792    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"]
Nov 03 01:07:09.604701 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:09.604669    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"]
Nov 03 01:07:12.462382 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:12.462338    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"]
Nov 03 01:07:12.492475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:12.492443    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"]
Nov 03 01:07:14.605766 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:14.605719    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" podUID="45af085b-efbd-4726-8640-8f32e1ee55a8" containerName="payload-processing" containerID="cri-o://4878b75c730000b71370a459a96269f67a67b9b65436ca6243238823281834b7" gracePeriod=25
Nov 03 01:07:14.727286 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:14.726043    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"]
Nov 03 01:07:14.748191 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:14.748149    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" podUID="b10511c0-de08-4d50-9be5-976d1b7e0887" containerName="maas-api" containerID="cri-o://fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242" gracePeriod=30
Nov 03 01:07:15.257198 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.257112    6000 generic.go:356] "Generic (PLEG): container finished" podID="b10511c0-de08-4d50-9be5-976d1b7e0887" containerID="fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242" exitCode=0
Nov 03 01:07:15.257635 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.257317    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 01:07:15.257797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.257339    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" event={"ID":"b10511c0-de08-4d50-9be5-976d1b7e0887","Type":"ContainerDied","Data":"fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242"}
Nov 03 01:07:15.257934 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.257916    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" event={"ID":"b10511c0-de08-4d50-9be5-976d1b7e0887","Type":"ContainerDied","Data":"bad3ca36894822c0e7eb58dc5e2f75d0a935cc998f2dc9659b3abb9d1d3488be"}
Nov 03 01:07:15.258172 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.257997    6000 scope.go:122] "RemoveContainer" containerID="fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242"
Nov 03 01:07:15.285753 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.285732    6000 scope.go:122] "RemoveContainer" containerID="fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242"
Nov 03 01:07:15.287572 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:15.287502    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242\": container with ID starting with fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242 not found: ID does not exist" containerID="fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242"
Nov 03 01:07:15.287572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.287544    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242"} err="failed to get container status \"fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242\": rpc error: code = NotFound desc = could not find container \"fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242\": container with ID starting with fd0257f6ef45627c996ff3fdc342e002108e782cb626813fef4b717271353242 not found: ID does not exist"
Nov 03 01:07:15.397005 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.396975    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b10511c0-de08-4d50-9be5-976d1b7e0887-kube-api-access-9vpdp\" (UniqueName: \"kubernetes.io/projected/b10511c0-de08-4d50-9be5-976d1b7e0887-kube-api-access-9vpdp\") pod \"b10511c0-de08-4d50-9be5-976d1b7e0887\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") "
Nov 03 01:07:15.397187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.397058    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls\") pod \"b10511c0-de08-4d50-9be5-976d1b7e0887\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") "
Nov 03 01:07:15.397187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.397170    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-metrics-certs\") pod \"b10511c0-de08-4d50-9be5-976d1b7e0887\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") "
Nov 03 01:07:15.397362 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.397280    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b10511c0-de08-4d50-9be5-976d1b7e0887-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10511c0-de08-4d50-9be5-976d1b7e0887-openshift-service-ca\") pod \"b10511c0-de08-4d50-9be5-976d1b7e0887\" (UID: \"b10511c0-de08-4d50-9be5-976d1b7e0887\") "
Nov 03 01:07:15.397755 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.397728    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b10511c0-de08-4d50-9be5-976d1b7e0887-openshift-service-ca" pod "b10511c0-de08-4d50-9be5-976d1b7e0887" (UID: "b10511c0-de08-4d50-9be5-976d1b7e0887"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:15.399720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.399693    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-metrics-certs" pod "b10511c0-de08-4d50-9be5-976d1b7e0887" (UID: "b10511c0-de08-4d50-9be5-976d1b7e0887"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:15.400034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.400009    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b10511c0-de08-4d50-9be5-976d1b7e0887-kube-api-access-9vpdp" pod "b10511c0-de08-4d50-9be5-976d1b7e0887" (UID: "b10511c0-de08-4d50-9be5-976d1b7e0887"). InnerVolumeSpecName "kube-api-access-9vpdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:15.400129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.400075    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls" pod "b10511c0-de08-4d50-9be5-976d1b7e0887" (UID: "b10511c0-de08-4d50-9be5-976d1b7e0887"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:15.498769 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.498732    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b10511c0-de08-4d50-9be5-976d1b7e0887-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:15.498769 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.498770    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vpdp\" (UniqueName: \"kubernetes.io/projected/b10511c0-de08-4d50-9be5-976d1b7e0887-kube-api-access-9vpdp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:15.499065 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.498810    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:15.499065 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:15.498825    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b10511c0-de08-4d50-9be5-976d1b7e0887-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:16.266646 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:16.266619    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"
Nov 03 01:07:16.331165 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:16.331013    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"]
Nov 03 01:07:16.342152 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:16.342115    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2"]
Nov 03 01:07:16.405293 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:16.405240    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" podUID="45af085b-efbd-4726-8640-8f32e1ee55a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:16.570208 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:16.570135    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b10511c0-de08-4d50-9be5-976d1b7e0887" path="/var/lib/kubelet/pods/b10511c0-de08-4d50-9be5-976d1b7e0887/volumes"
Nov 03 01:07:17.466423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:17.466331    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" podUID="82f00888-aaea-4d21-a0f4-dabaedf20c74" containerName="payload-processing" containerID="cri-o://a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7" gracePeriod=25
Nov 03 01:07:17.494390 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:17.494348    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" podUID="68389007-bcab-4bf9-a05a-dd6e469baefe" containerName="payload-pre-processing" containerID="cri-o://65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9" gracePeriod=25
Nov 03 01:07:18.024596 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.024553    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-8ed55a-845b4df9b4-jg5v2" podUID="b10511c0-de08-4d50-9be5-976d1b7e0887" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.1.15:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:07:18.578376 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.578340    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn" podUID="b703fed3-ecc3-44fc-9275-0831d71e9e1c" containerName="main" containerID="cri-o://74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15" gracePeriod=45
Nov 03 01:07:18.883104 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.883079    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:07:18.956866 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.956825    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-model-cache\") pod \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") "
Nov 03 01:07:18.957078 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.956916    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tls-certs\" (UniqueName: \"kubernetes.io/secret/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tls-certs\") pod \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") "
Nov 03 01:07:18.957078 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.956956    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-home\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-home\") pod \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") "
Nov 03 01:07:18.957204 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.957091    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-dshm\") pod \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") "
Nov 03 01:07:18.957204 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.957141    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tmp-dir\") pod \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") "
Nov 03 01:07:18.957340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.957247    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b703fed3-ecc3-44fc-9275-0831d71e9e1c-kube-api-access-jtp79\" (UniqueName: \"kubernetes.io/projected/b703fed3-ecc3-44fc-9275-0831d71e9e1c-kube-api-access-jtp79\") pod \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\" (UID: \"b703fed3-ecc3-44fc-9275-0831d71e9e1c\") "
Nov 03 01:07:18.958736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.958706    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tmp-dir" pod "b703fed3-ecc3-44fc-9275-0831d71e9e1c" (UID: "b703fed3-ecc3-44fc-9275-0831d71e9e1c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:18.960004 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.958781    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-model-cache" pod "b703fed3-ecc3-44fc-9275-0831d71e9e1c" (UID: "b703fed3-ecc3-44fc-9275-0831d71e9e1c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:18.960004 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.959053    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-home" pod "b703fed3-ecc3-44fc-9275-0831d71e9e1c" (UID: "b703fed3-ecc3-44fc-9275-0831d71e9e1c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:18.962093 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.962055    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-dshm" pod "b703fed3-ecc3-44fc-9275-0831d71e9e1c" (UID: "b703fed3-ecc3-44fc-9275-0831d71e9e1c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:18.962495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.962475    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tls-certs" pod "b703fed3-ecc3-44fc-9275-0831d71e9e1c" (UID: "b703fed3-ecc3-44fc-9275-0831d71e9e1c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:07:18.962704 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:18.962686    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b703fed3-ecc3-44fc-9275-0831d71e9e1c-kube-api-access-jtp79" pod "b703fed3-ecc3-44fc-9275-0831d71e9e1c" (UID: "b703fed3-ecc3-44fc-9275-0831d71e9e1c"). InnerVolumeSpecName "kube-api-access-jtp79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:19.059533 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.059502    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.059533 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.059530    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.059533 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.059540    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.059772 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.059548    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.059772 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.059556    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b703fed3-ecc3-44fc-9275-0831d71e9e1c-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.059772 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.059565    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtp79\" (UniqueName: \"kubernetes.io/projected/b703fed3-ecc3-44fc-9275-0831d71e9e1c-kube-api-access-jtp79\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:19.290088 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.290057    6000 generic.go:356] "Generic (PLEG): container finished" podID="b703fed3-ecc3-44fc-9275-0831d71e9e1c" containerID="74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15" exitCode=0
Nov 03 01:07:19.290371 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.290172    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"
Nov 03 01:07:19.290371 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.290209    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn" event={"ID":"b703fed3-ecc3-44fc-9275-0831d71e9e1c","Type":"ContainerDied","Data":"74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15"}
Nov 03 01:07:19.290371 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.290274    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn" event={"ID":"b703fed3-ecc3-44fc-9275-0831d71e9e1c","Type":"ContainerDied","Data":"3d09c1f52f2ca4a13e59e57616528b9760ba6b0597175cc30bb3b6981106b592"}
Nov 03 01:07:19.290371 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.290298    6000 scope.go:122] "RemoveContainer" containerID="74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15"
Nov 03 01:07:19.302127 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.302097    6000 scope.go:122] "RemoveContainer" containerID="74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15"
Nov 03 01:07:19.302409 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:19.302391    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15\": container with ID starting with 74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15 not found: ID does not exist" containerID="74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15"
Nov 03 01:07:19.302508 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.302418    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15"} err="failed to get container status \"74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15\": rpc error: code = NotFound desc = could not find container \"74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15\": container with ID starting with 74e915217628ccdd48e6632258055f4bcb8d60f9fe15a1531b8b9ea4a1d12d15 not found: ID does not exist"
Nov 03 01:07:19.339562 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.339536    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"]
Nov 03 01:07:19.350336 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.350315    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-70c7d6/test-model-70c7d6-kserve-69486c8fc7-8smqn"]
Nov 03 01:07:19.596938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.596848    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"]
Nov 03 01:07:19.597497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:19.597113    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" containerID="cri-o://107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47" gracePeriod=30
Nov 03 01:07:19.906144 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:19.906104    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:07:20.569163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:20.569128    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b703fed3-ecc3-44fc-9275-0831d71e9e1c" path="/var/lib/kubelet/pods/b703fed3-ecc3-44fc-9275-0831d71e9e1c/volumes"
Nov 03 01:07:22.867453 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:22.867411    6000 patch_prober.go:29] interesting pod/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:07:22.868138 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:22.867473    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:07:24.181618 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:24.181582    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:07:24.386538 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:24.386498    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" podUID="82f00888-aaea-4d21-a0f4-dabaedf20c74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:24.386735 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:24.386554    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" podUID="68389007-bcab-4bf9-a05a-dd6e469baefe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:24.916934 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:24.916906    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 01:07:25.018939 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.018895    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-certs\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019158 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.018991    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-envoy\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019158 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019083    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-podinfo\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019158 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019135    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-kube-api-access-s8jf6\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-kube-api-access-s8jf6\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019199    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-data\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019449 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019407    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-socket\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019476    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-token\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-token\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019197    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-certs" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:25.019664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019611    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-credential-socket\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019647    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istiod-ca-cert\") pod \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\" (UID: \"14e692c0-72c9-4292-ae3b-454ff2d5a25f\") "
Nov 03 01:07:25.019794 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.019737    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-data" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:25.020095 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.020053    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-credential-socket" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:25.020215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.020100    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-socket" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:25.020329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.020284    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.020329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.020302    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.020329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.020312    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.020329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.020322    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.020560 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.020409    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istiod-ca-cert" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:25.022352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.022315    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-envoy" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:07:25.022872 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.022836    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-podinfo" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:07:25.023641 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.023603    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-token" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:25.024101 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.024077    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-kube-api-access-s8jf6" pod "14e692c0-72c9-4292-ae3b-454ff2d5a25f" (UID: "14e692c0-72c9-4292-ae3b-454ff2d5a25f"). InnerVolumeSpecName "kube-api-access-s8jf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:25.124768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.123550    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.125173 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.125152    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.125313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.125297    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8jf6\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-kube-api-access-s8jf6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.125432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.125419    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.125519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.125509    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14e692c0-72c9-4292-ae3b-454ff2d5a25f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:25.356402 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.356363    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"
Nov 03 01:07:25.356932 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.356353    6000 generic.go:356] "Generic (PLEG): container finished" podID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerID="107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47" exitCode=0
Nov 03 01:07:25.356932 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.356359    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" event={"ID":"14e692c0-72c9-4292-ae3b-454ff2d5a25f","Type":"ContainerDied","Data":"107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47"}
Nov 03 01:07:25.356932 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.356721    6000 scope.go:122] "RemoveContainer" containerID="107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47"
Nov 03 01:07:25.357102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.356591    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr" event={"ID":"14e692c0-72c9-4292-ae3b-454ff2d5a25f","Type":"ContainerDied","Data":"357296bb81c0f0d8ace7dbc51fb78027e72be6338ae9a9868c6bd55ad5021bb2"}
Nov 03 01:07:25.380040 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.380016    6000 scope.go:122] "RemoveContainer" containerID="a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d"
Nov 03 01:07:25.392920 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.392893    6000 scope.go:122] "RemoveContainer" containerID="107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47"
Nov 03 01:07:25.393302 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:25.393248    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47\": container with ID starting with 107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47 not found: ID does not exist" containerID="107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47"
Nov 03 01:07:25.393417 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.393303    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47"} err="failed to get container status \"107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47\": rpc error: code = NotFound desc = could not find container \"107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47\": container with ID starting with 107abd5c6ae9a741bdb3253553005a0c1563644fddbe61e2bd021956037e0e47 not found: ID does not exist"
Nov 03 01:07:25.393417 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.393321    6000 scope.go:122] "RemoveContainer" containerID="a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d"
Nov 03 01:07:25.393620 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:25.393593    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d\": container with ID starting with a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d not found: ID does not exist" containerID="a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d"
Nov 03 01:07:25.393693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.393630    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d"} err="failed to get container status \"a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d\": rpc error: code = NotFound desc = could not find container \"a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d\": container with ID starting with a466e971e3c04dd261fe50923e1c6b8c21ed2ab7ef6ba8b23d5e44925c2d500d not found: ID does not exist"
Nov 03 01:07:25.414505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.414479    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"]
Nov 03 01:07:25.419424 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:25.419393    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-29d09b-openshift-default-56b588f5c5-mmndr"]
Nov 03 01:07:26.408128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:26.408091    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" podUID="45af085b-efbd-4726-8640-8f32e1ee55a8" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:26.642029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:26.641496    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" path="/var/lib/kubelet/pods/14e692c0-72c9-4292-ae3b-454ff2d5a25f/volumes"
Nov 03 01:07:34.386434 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.386390    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" podUID="68389007-bcab-4bf9-a05a-dd6e469baefe" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:34.386967 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.386941    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" podUID="82f00888-aaea-4d21-a0f4-dabaedf20c74" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:07:34.811761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.811718    6000 generic.go:356] "Generic (PLEG): container finished" podID="45af085b-efbd-4726-8640-8f32e1ee55a8" containerID="4878b75c730000b71370a459a96269f67a67b9b65436ca6243238823281834b7" exitCode=0
Nov 03 01:07:34.811928 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.811794    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" event={"ID":"45af085b-efbd-4726-8640-8f32e1ee55a8","Type":"ContainerDied","Data":"4878b75c730000b71370a459a96269f67a67b9b65436ca6243238823281834b7"}
Nov 03 01:07:34.903207 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.903172    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 01:07:34.958637 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.958608    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45af085b-efbd-4726-8640-8f32e1ee55a8-kube-api-access-jkhdv\" (UniqueName: \"kubernetes.io/projected/45af085b-efbd-4726-8640-8f32e1ee55a8-kube-api-access-jkhdv\") pod \"45af085b-efbd-4726-8640-8f32e1ee55a8\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") "
Nov 03 01:07:34.958867 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.958843    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45af085b-efbd-4726-8640-8f32e1ee55a8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45af085b-efbd-4726-8640-8f32e1ee55a8-plugins-config-volume\") pod \"45af085b-efbd-4726-8640-8f32e1ee55a8\" (UID: \"45af085b-efbd-4726-8640-8f32e1ee55a8\") "
Nov 03 01:07:34.959287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.959240    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45af085b-efbd-4726-8640-8f32e1ee55a8-plugins-config-volume" pod "45af085b-efbd-4726-8640-8f32e1ee55a8" (UID: "45af085b-efbd-4726-8640-8f32e1ee55a8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:34.959549 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.959493    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45af085b-efbd-4726-8640-8f32e1ee55a8-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:34.961168 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:34.961144    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45af085b-efbd-4726-8640-8f32e1ee55a8-kube-api-access-jkhdv" pod "45af085b-efbd-4726-8640-8f32e1ee55a8" (UID: "45af085b-efbd-4726-8640-8f32e1ee55a8"). InnerVolumeSpecName "kube-api-access-jkhdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:35.061223 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:35.061131    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkhdv\" (UniqueName: \"kubernetes.io/projected/45af085b-efbd-4726-8640-8f32e1ee55a8-kube-api-access-jkhdv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:35.821420 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:35.821388    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"
Nov 03 01:07:35.821955 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:35.821412    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq" event={"ID":"45af085b-efbd-4726-8640-8f32e1ee55a8","Type":"ContainerDied","Data":"e6b449c0245c4ab60bb1017771e966222d92f7a736ddf895938bcec4e55aeb40"}
Nov 03 01:07:35.821955 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:35.821491    6000 scope.go:122] "RemoveContainer" containerID="4878b75c730000b71370a459a96269f67a67b9b65436ca6243238823281834b7"
Nov 03 01:07:35.871058 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:35.871029    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"]
Nov 03 01:07:35.882460 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:35.882431    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-8ed55a-646db6d969-nx6hq"]
Nov 03 01:07:36.569062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:36.569028    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45af085b-efbd-4726-8640-8f32e1ee55a8" path="/var/lib/kubelet/pods/45af085b-efbd-4726-8640-8f32e1ee55a8/volumes"
Nov 03 01:07:37.804876 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.804854    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 01:07:37.823829 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.823809    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 01:07:37.838865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.838834    6000 generic.go:356] "Generic (PLEG): container finished" podID="68389007-bcab-4bf9-a05a-dd6e469baefe" containerID="65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9" exitCode=0
Nov 03 01:07:37.839010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.838877    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"
Nov 03 01:07:37.839010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.838911    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" event={"ID":"68389007-bcab-4bf9-a05a-dd6e469baefe","Type":"ContainerDied","Data":"65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9"}
Nov 03 01:07:37.839145 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.839064    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65" event={"ID":"68389007-bcab-4bf9-a05a-dd6e469baefe","Type":"ContainerDied","Data":"90390a70237d463329bf7292d77dd07de4fc8515499cad6655199e28a3ef00b8"}
Nov 03 01:07:37.839145 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.839103    6000 scope.go:122] "RemoveContainer" containerID="65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9"
Nov 03 01:07:37.840661 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.840641    6000 generic.go:356] "Generic (PLEG): container finished" podID="82f00888-aaea-4d21-a0f4-dabaedf20c74" containerID="a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7" exitCode=0
Nov 03 01:07:37.840783 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.840682    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"
Nov 03 01:07:37.840783 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.840773    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" event={"ID":"82f00888-aaea-4d21-a0f4-dabaedf20c74","Type":"ContainerDied","Data":"a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7"}
Nov 03 01:07:37.840895 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.840814    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd" event={"ID":"82f00888-aaea-4d21-a0f4-dabaedf20c74","Type":"ContainerDied","Data":"2abf9b195ee8d8a35221cecf7de3fac748378ba9741f593e1004fe9e636fcd22"}
Nov 03 01:07:37.852600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.852579    6000 scope.go:122] "RemoveContainer" containerID="65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9"
Nov 03 01:07:37.852957 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:37.852933    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9\": container with ID starting with 65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9 not found: ID does not exist" containerID="65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9"
Nov 03 01:07:37.853040 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.852967    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9"} err="failed to get container status \"65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9\": rpc error: code = NotFound desc = could not find container \"65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9\": container with ID starting with 65d22f653ab5ee9982b197cc4a9c268e289c31d444d673cd5370b8fa87083db9 not found: ID does not exist"
Nov 03 01:07:37.853040 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.852984    6000 scope.go:122] "RemoveContainer" containerID="a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7"
Nov 03 01:07:37.864857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.864832    6000 scope.go:122] "RemoveContainer" containerID="a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7"
Nov 03 01:07:37.865219 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:37.865193    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7\": container with ID starting with a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7 not found: ID does not exist" containerID="a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7"
Nov 03 01:07:37.865358 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.865232    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7"} err="failed to get container status \"a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7\": rpc error: code = NotFound desc = could not find container \"a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7\": container with ID starting with a7608a69f535aaa1373903aba220346677be2ef06caf8806b94c3b325d4faaa7 not found: ID does not exist"
Nov 03 01:07:37.913141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.913103    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68389007-bcab-4bf9-a05a-dd6e469baefe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68389007-bcab-4bf9-a05a-dd6e469baefe-plugins-config-volume\") pod \"68389007-bcab-4bf9-a05a-dd6e469baefe\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") "
Nov 03 01:07:37.913359 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.913293    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f00888-aaea-4d21-a0f4-dabaedf20c74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f00888-aaea-4d21-a0f4-dabaedf20c74-plugins-config-volume\") pod \"82f00888-aaea-4d21-a0f4-dabaedf20c74\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") "
Nov 03 01:07:37.913359 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.913339    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68389007-bcab-4bf9-a05a-dd6e469baefe-kube-api-access-vgj2c\" (UniqueName: \"kubernetes.io/projected/68389007-bcab-4bf9-a05a-dd6e469baefe-kube-api-access-vgj2c\") pod \"68389007-bcab-4bf9-a05a-dd6e469baefe\" (UID: \"68389007-bcab-4bf9-a05a-dd6e469baefe\") "
Nov 03 01:07:37.913516 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.913381    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f00888-aaea-4d21-a0f4-dabaedf20c74-kube-api-access-2l6kv\" (UniqueName: \"kubernetes.io/projected/82f00888-aaea-4d21-a0f4-dabaedf20c74-kube-api-access-2l6kv\") pod \"82f00888-aaea-4d21-a0f4-dabaedf20c74\" (UID: \"82f00888-aaea-4d21-a0f4-dabaedf20c74\") "
Nov 03 01:07:37.913613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.913574    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68389007-bcab-4bf9-a05a-dd6e469baefe-plugins-config-volume" pod "68389007-bcab-4bf9-a05a-dd6e469baefe" (UID: "68389007-bcab-4bf9-a05a-dd6e469baefe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:37.913680 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.913651    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f00888-aaea-4d21-a0f4-dabaedf20c74-plugins-config-volume" pod "82f00888-aaea-4d21-a0f4-dabaedf20c74" (UID: "82f00888-aaea-4d21-a0f4-dabaedf20c74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:07:37.914235 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.914211    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68389007-bcab-4bf9-a05a-dd6e469baefe-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.914353 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.914240    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/82f00888-aaea-4d21-a0f4-dabaedf20c74-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:37.915938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.915910    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f00888-aaea-4d21-a0f4-dabaedf20c74-kube-api-access-2l6kv" pod "82f00888-aaea-4d21-a0f4-dabaedf20c74" (UID: "82f00888-aaea-4d21-a0f4-dabaedf20c74"). InnerVolumeSpecName "kube-api-access-2l6kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:37.916406 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:37.916386    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68389007-bcab-4bf9-a05a-dd6e469baefe-kube-api-access-vgj2c" pod "68389007-bcab-4bf9-a05a-dd6e469baefe" (UID: "68389007-bcab-4bf9-a05a-dd6e469baefe"). InnerVolumeSpecName "kube-api-access-vgj2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:07:38.015750 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.015665    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgj2c\" (UniqueName: \"kubernetes.io/projected/68389007-bcab-4bf9-a05a-dd6e469baefe-kube-api-access-vgj2c\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:38.015750 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.015698    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l6kv\" (UniqueName: \"kubernetes.io/projected/82f00888-aaea-4d21-a0f4-dabaedf20c74-kube-api-access-2l6kv\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:07:38.193830 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.193795    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"]
Nov 03 01:07:38.204310 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.204280    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-29d09b-7ccd86d88f-z59wd"]
Nov 03 01:07:38.236360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.236328    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"]
Nov 03 01:07:38.241330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.241277    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-29d09b-7b5fb8444c-ndv65"]
Nov 03 01:07:38.569264 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.569220    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68389007-bcab-4bf9-a05a-dd6e469baefe" path="/var/lib/kubelet/pods/68389007-bcab-4bf9-a05a-dd6e469baefe/volumes"
Nov 03 01:07:38.569693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:38.569679    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f00888-aaea-4d21-a0f4-dabaedf20c74" path="/var/lib/kubelet/pods/82f00888-aaea-4d21-a0f4-dabaedf20c74/volumes"
Nov 03 01:07:40.886734 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.886696    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"]
Nov 03 01:07:40.888557 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888527    6000 cpu_manager.go:414] "containerMap: removing container" podUID="26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" containerName="payload-processing"
Nov 03 01:07:40.888557 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888556    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" containerName="payload-processing"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888592    6000 cpu_manager.go:414] "containerMap: removing container" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888601    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888640    6000 cpu_manager.go:414] "containerMap: removing container" podUID="68389007-bcab-4bf9-a05a-dd6e469baefe" containerName="payload-pre-processing"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888649    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68389007-bcab-4bf9-a05a-dd6e469baefe" containerName="payload-pre-processing"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888670    6000 cpu_manager.go:414] "containerMap: removing container" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888678    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888722    6000 cpu_manager.go:414] "containerMap: removing container" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888729    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888744    6000 cpu_manager.go:414] "containerMap: removing container" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888751    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888765    6000 cpu_manager.go:414] "containerMap: removing container" podUID="82f00888-aaea-4d21-a0f4-dabaedf20c74" containerName="payload-processing"
Nov 03 01:07:40.888775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888772    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f00888-aaea-4d21-a0f4-dabaedf20c74" containerName="payload-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888798    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b10511c0-de08-4d50-9be5-976d1b7e0887" containerName="maas-api"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888806    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b10511c0-de08-4d50-9be5-976d1b7e0887" containerName="maas-api"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888826    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888833    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888866    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888874    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888900    6000 cpu_manager.go:414] "containerMap: removing container" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888908    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888926    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e049c769-a863-4f07-8100-3d5d62c681d3" containerName="payload-pre-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888932    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e049c769-a863-4f07-8100-3d5d62c681d3" containerName="payload-pre-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888942    6000 cpu_manager.go:414] "containerMap: removing container" podUID="997abdcd-62e3-489c-9150-653c4573d8e3" containerName="main"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888956    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="997abdcd-62e3-489c-9150-653c4573d8e3" containerName="main"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888987    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f7529e5d-44c6-4282-b229-a977f6377256" containerName="payload-pre-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.888995    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7529e5d-44c6-4282-b229-a977f6377256" containerName="payload-pre-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889016    6000 cpu_manager.go:414] "containerMap: removing container" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889023    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889045    6000 cpu_manager.go:414] "containerMap: removing container" podUID="3fc532fe-49e8-404d-ab30-fd29bae9f031" containerName="payload-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889052    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fc532fe-49e8-404d-ab30-fd29bae9f031" containerName="payload-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889060    6000 cpu_manager.go:414] "containerMap: removing container" podUID="45af085b-efbd-4726-8640-8f32e1ee55a8" containerName="payload-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889067    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45af085b-efbd-4726-8640-8f32e1ee55a8" containerName="payload-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889098    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" containerName="payload-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889104    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" containerName="payload-processing"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889118    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6a83d623-1d5f-4dfa-8b5a-f826ec572c44" containerName="main"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889124    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a83d623-1d5f-4dfa-8b5a-f826ec572c44" containerName="main"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889139    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b703fed3-ecc3-44fc-9275-0831d71e9e1c" containerName="main"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889147    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b703fed3-ecc3-44fc-9275-0831d71e9e1c" containerName="main"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889168    6000 cpu_manager.go:414] "containerMap: removing container" podUID="08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" containerName="maas-api"
Nov 03 01:07:40.889354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889175    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" containerName="maas-api"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889511    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889539    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7529e5d-44c6-4282-b229-a977f6377256" containerName="payload-pre-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889550    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="dda4f231-46f1-4753-bb41-d5e8745dd7d5" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889568    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fc532fe-49e8-404d-ab30-fd29bae9f031" containerName="payload-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889586    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="45af085b-efbd-4726-8640-8f32e1ee55a8" containerName="payload-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889603    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ce6bc1e-3ff5-4d18-b61e-0ac0243c55b4" containerName="payload-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889614    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a83d623-1d5f-4dfa-8b5a-f826ec572c44" containerName="main"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889627    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b703fed3-ecc3-44fc-9275-0831d71e9e1c" containerName="main"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889635    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889650    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ba31f7-4aa6-46d1-bd29-3ab667ddaa5f" containerName="maas-api"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889660    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="26767d83-e7c7-444c-bc3c-b9e4a71bc8d1" containerName="payload-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889670    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="a5f12ebd-dce0-4c4e-920b-c260acdb3b17" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889695    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="68389007-bcab-4bf9-a05a-dd6e469baefe" containerName="payload-pre-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889704    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889722    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="356bea3c-a365-4f7a-81f5-35c9bd90b427" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889740    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f00888-aaea-4d21-a0f4-dabaedf20c74" containerName="payload-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889748    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b10511c0-de08-4d50-9be5-976d1b7e0887" containerName="maas-api"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889755    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b63488ad-6f80-48ed-8b9e-66ca54e9fff1" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889771    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1273c82d-11d9-4a6a-aff7-a190d9e419d5" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889781    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="06270d58-ffc7-4d70-afac-7f8f99570583" containerName="istio-proxy"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889795    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e049c769-a863-4f07-8100-3d5d62c681d3" containerName="payload-pre-processing"
Nov 03 01:07:40.890740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.889812    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="997abdcd-62e3-489c-9150-653c4573d8e3" containerName="main"
Nov 03 01:07:40.907951 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.907916    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:40.909960 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.909922    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"]
Nov 03 01:07:40.921954 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.921915    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"kube-root-ca.crt\""
Nov 03 01:07:40.933512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.933483    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"openshift-service-ca.crt\""
Nov 03 01:07:40.933658 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.933587    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"default-dockercfg-8kwzh\""
Nov 03 01:07:40.933658 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.933595    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"rate-test-model-9908e8-kserve-self-signed-certs\""
Nov 03 01:07:40.969712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.969680    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-model-cache\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:40.969864 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.969728    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-tmp-dir\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:40.969864 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.969810    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-home\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:40.970295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.969900    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpglf\" (UniqueName: \"kubernetes.io/projected/94c479c0-25ca-49b6-b99d-55a39d17cc52-kube-api-access-fpglf\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:40.970295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.969951    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-dshm\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:40.970295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:40.970101    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c479c0-25ca-49b6-b99d-55a39d17cc52-tls-certs\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.071489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.071454    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c479c0-25ca-49b6-b99d-55a39d17cc52-tls-certs\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.071663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.071507    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-model-cache\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.071663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.071539    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-tmp-dir\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.071663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.071581    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-home\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.071663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.071636    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpglf\" (UniqueName: \"kubernetes.io/projected/94c479c0-25ca-49b6-b99d-55a39d17cc52-kube-api-access-fpglf\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.071905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.071732    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-dshm\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.071998 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.071981    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-model-cache\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.072063 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.072039    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-home\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.072117 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.072077    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-tmp-dir\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.074100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.074077    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-dshm\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.074335 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.074313    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c479c0-25ca-49b6-b99d-55a39d17cc52-tls-certs\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.089988 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.089965    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpglf\" (UniqueName: \"kubernetes.io/projected/94c479c0-25ca-49b6-b99d-55a39d17cc52-kube-api-access-fpglf\") pod \"rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.225082 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.225005    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:41.540607 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:07:41.540577    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94c479c0_25ca_49b6_b99d_55a39d17cc52.slice/crio-c4e1eeb43b08b10c50b8e1672ba2c23a183a8c29a086ef111f1b7a4a483c92d3 WatchSource:0}: Error finding container c4e1eeb43b08b10c50b8e1672ba2c23a183a8c29a086ef111f1b7a4a483c92d3: Status 404 returned error can't find the container with id c4e1eeb43b08b10c50b8e1672ba2c23a183a8c29a086ef111f1b7a4a483c92d3
Nov 03 01:07:41.540953 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.540926    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"]
Nov 03 01:07:41.879027 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.878986    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk" event={"ID":"94c479c0-25ca-49b6-b99d-55a39d17cc52","Type":"ContainerStarted","Data":"60377d499bf149c9a8f20caff2df16103a6ec643145663af57d7b2a0d47832f0"}
Nov 03 01:07:41.879210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.879039    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk" event={"ID":"94c479c0-25ca-49b6-b99d-55a39d17cc52","Type":"ContainerStarted","Data":"c4e1eeb43b08b10c50b8e1672ba2c23a183a8c29a086ef111f1b7a4a483c92d3"}
Nov 03 01:07:41.918144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:41.918091    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk" podStartSLOduration=1.918075961 podStartE2EDuration="1.918075961s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:07:41.916204571 +0000 UTC m=+3699.471556596" watchObservedRunningTime="2026-11-03 01:07:41.918075961 +0000 UTC m=+3699.473427985"
Nov 03 01:07:42.225874 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:42.225771    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:49.897225 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:07:49.897189    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:07:51.225720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:51.225684    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:51.239286 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:51.239244    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:07:51.969192 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:07:51.969161    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:08:05.992658 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:05.992621    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"]
Nov 03 01:08:05.993487 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:05.993469    6000 cpu_manager.go:414] "containerMap: removing container" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy"
Nov 03 01:08:05.993545 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:05.993488    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy"
Nov 03 01:08:05.993580 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:05.993546    6000 cpu_manager.go:414] "containerMap: removing container" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy"
Nov 03 01:08:05.993580 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:05.993551    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14e692c0-72c9-4292-ae3b-454ff2d5a25f" containerName="istio-proxy"
Nov 03 01:08:05.993675 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:05.993666    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fda859-71bf-4b39-bb5c-5a26bd7ef43a" containerName="istio-proxy"
Nov 03 01:08:05.997326 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:05.997306    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.014901 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.014846    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"]
Nov 03 01:08:06.021885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.021857    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-c6e808d7\""
Nov 03 01:08:06.022033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.021945    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-c6e808d7-dockercfg-hmn48\""
Nov 03 01:08:06.058701 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.058666    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"]
Nov 03 01:08:06.063249 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.063224    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.085567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.085541    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"]
Nov 03 01:08:06.154293 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.152427    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-429z6\" (UniqueName: \"kubernetes.io/projected/30a5aa39-82cd-479f-a09b-cab7db69e4ae-kube-api-access-429z6\") pod \"payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") " pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.154293 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.152692    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5aa39-82cd-479f-a09b-cab7db69e4ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") " pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.254357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.254152    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-429z6\" (UniqueName: \"kubernetes.io/projected/30a5aa39-82cd-479f-a09b-cab7db69e4ae-kube-api-access-429z6\") pod \"payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") " pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.254592 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.254488    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jbbx\" (UniqueName: \"kubernetes.io/projected/69a516d6-a843-40be-9449-53a5cb69376d-kube-api-access-6jbbx\") pod \"payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.254592 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.254526    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a516d6-a843-40be-9449-53a5cb69376d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.254720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.254668    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5aa39-82cd-479f-a09b-cab7db69e4ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") " pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.258317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.255880    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5aa39-82cd-479f-a09b-cab7db69e4ae-plugins-config-volume\") pod \"payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") " pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.281312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.275951    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-429z6\" (UniqueName: \"kubernetes.io/projected/30a5aa39-82cd-479f-a09b-cab7db69e4ae-kube-api-access-429z6\") pod \"payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") " pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.312144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.312107    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:06.356286 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.356229    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jbbx\" (UniqueName: \"kubernetes.io/projected/69a516d6-a843-40be-9449-53a5cb69376d-kube-api-access-6jbbx\") pod \"payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.356466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.356304    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a516d6-a843-40be-9449-53a5cb69376d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.357000 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.356978    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a516d6-a843-40be-9449-53a5cb69376d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.378505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.378475    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jbbx\" (UniqueName: \"kubernetes.io/projected/69a516d6-a843-40be-9449-53a5cb69376d-kube-api-access-6jbbx\") pod \"payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.620213 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:08:06.620184    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30a5aa39_82cd_479f_a09b_cab7db69e4ae.slice/crio-6f4cbbaa4369c9d04b77ce5b1772ff88a2c0403c6a79ba44cd9d67df97a5202f WatchSource:0}: Error finding container 6f4cbbaa4369c9d04b77ce5b1772ff88a2c0403c6a79ba44cd9d67df97a5202f: Status 404 returned error can't find the container with id 6f4cbbaa4369c9d04b77ce5b1772ff88a2c0403c6a79ba44cd9d67df97a5202f
Nov 03 01:08:06.621528 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.621501    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"]
Nov 03 01:08:06.678497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.678469    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:06.982198 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:06.982170    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"]
Nov 03 01:08:06.982901 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:08:06.982864    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a516d6_a843_40be_9449_53a5cb69376d.slice/crio-83c812f3452cb9e4f1b2be4d19c0c7e33e020b8f929ce9fcf2e9fa802828ad30 WatchSource:0}: Error finding container 83c812f3452cb9e4f1b2be4d19c0c7e33e020b8f929ce9fcf2e9fa802828ad30: Status 404 returned error can't find the container with id 83c812f3452cb9e4f1b2be4d19c0c7e33e020b8f929ce9fcf2e9fa802828ad30
Nov 03 01:08:07.094264 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.094227    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk" event={"ID":"30a5aa39-82cd-479f-a09b-cab7db69e4ae","Type":"ContainerStarted","Data":"8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f"}
Nov 03 01:08:07.094694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.094302    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk" event={"ID":"30a5aa39-82cd-479f-a09b-cab7db69e4ae","Type":"ContainerStarted","Data":"6f4cbbaa4369c9d04b77ce5b1772ff88a2c0403c6a79ba44cd9d67df97a5202f"}
Nov 03 01:08:07.094694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.094338    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:07.096089 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.096060    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl" event={"ID":"69a516d6-a843-40be-9449-53a5cb69376d","Type":"ContainerStarted","Data":"e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364"}
Nov 03 01:08:07.096209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.096118    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:07.096209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.096143    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl" event={"ID":"69a516d6-a843-40be-9449-53a5cb69376d","Type":"ContainerStarted","Data":"83c812f3452cb9e4f1b2be4d19c0c7e33e020b8f929ce9fcf2e9fa802828ad30"}
Nov 03 01:08:07.132615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.132568    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk" podStartSLOduration=2.132549349 podStartE2EDuration="2.132549349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:07.13233004 +0000 UTC m=+3724.687682068" watchObservedRunningTime="2026-11-03 01:08:07.132549349 +0000 UTC m=+3724.687901373"
Nov 03 01:08:07.171169 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:07.171117    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl" podStartSLOduration=1.171103482 podStartE2EDuration="1.171103482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:07.168412542 +0000 UTC m=+3724.723764568" watchObservedRunningTime="2026-11-03 01:08:07.171103482 +0000 UTC m=+3724.726455545"
Nov 03 01:08:08.818038 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:08.818004    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"]
Nov 03 01:08:08.818690 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:08.818211    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46" podUID="28473950-3c2f-4dbc-be97-6c05b5efced2" containerName="maas-api" containerID="cri-o://15cbf560f5227e06d19cc8bdad49771dd2f787df82e109ec1cec877a345aa30b" gracePeriod=30
Nov 03 01:08:09.147276 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.146728    6000 generic.go:356] "Generic (PLEG): container finished" podID="28473950-3c2f-4dbc-be97-6c05b5efced2" containerID="15cbf560f5227e06d19cc8bdad49771dd2f787df82e109ec1cec877a345aa30b" exitCode=0
Nov 03 01:08:09.147276 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.147017    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46" event={"ID":"28473950-3c2f-4dbc-be97-6c05b5efced2","Type":"ContainerDied","Data":"15cbf560f5227e06d19cc8bdad49771dd2f787df82e109ec1cec877a345aa30b"}
Nov 03 01:08:09.147276 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.147081    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46" event={"ID":"28473950-3c2f-4dbc-be97-6c05b5efced2","Type":"ContainerDied","Data":"24d8b9123a68cd5c4dea60a63f694658b11e498fc0a72f4df04f8ef3a19a03ba"}
Nov 03 01:08:09.147276 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.147108    6000 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="24d8b9123a68cd5c4dea60a63f694658b11e498fc0a72f4df04f8ef3a19a03ba"
Nov 03 01:08:09.154223 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.150683    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:08:09.219785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.210177    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-maas-api-tls\") pod \"28473950-3c2f-4dbc-be97-6c05b5efced2\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") "
Nov 03 01:08:09.253283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.226636    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28473950-3c2f-4dbc-be97-6c05b5efced2-kube-api-access-v88pz\" (UniqueName: \"kubernetes.io/projected/28473950-3c2f-4dbc-be97-6c05b5efced2-kube-api-access-v88pz\") pod \"28473950-3c2f-4dbc-be97-6c05b5efced2\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") "
Nov 03 01:08:09.253746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.253717    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28473950-3c2f-4dbc-be97-6c05b5efced2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28473950-3c2f-4dbc-be97-6c05b5efced2-openshift-service-ca\") pod \"28473950-3c2f-4dbc-be97-6c05b5efced2\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") "
Nov 03 01:08:09.254359 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.254210    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-metrics-certs\") pod \"28473950-3c2f-4dbc-be97-6c05b5efced2\" (UID: \"28473950-3c2f-4dbc-be97-6c05b5efced2\") "
Nov 03 01:08:09.328281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.257015    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28473950-3c2f-4dbc-be97-6c05b5efced2-openshift-service-ca" pod "28473950-3c2f-4dbc-be97-6c05b5efced2" (UID: "28473950-3c2f-4dbc-be97-6c05b5efced2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:08:09.328281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.257133    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-maas-api-tls" pod "28473950-3c2f-4dbc-be97-6c05b5efced2" (UID: "28473950-3c2f-4dbc-be97-6c05b5efced2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:08:09.328281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.257681    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28473950-3c2f-4dbc-be97-6c05b5efced2-kube-api-access-v88pz" pod "28473950-3c2f-4dbc-be97-6c05b5efced2" (UID: "28473950-3c2f-4dbc-be97-6c05b5efced2"). InnerVolumeSpecName "kube-api-access-v88pz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:08:09.328281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.260043    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-metrics-certs" pod "28473950-3c2f-4dbc-be97-6c05b5efced2" (UID: "28473950-3c2f-4dbc-be97-6c05b5efced2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:08:09.421383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.421285    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v88pz\" (UniqueName: \"kubernetes.io/projected/28473950-3c2f-4dbc-be97-6c05b5efced2-kube-api-access-v88pz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:08:09.421383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.421317    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28473950-3c2f-4dbc-be97-6c05b5efced2-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:08:09.421383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.421332    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:08:09.421383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:09.421346    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28473950-3c2f-4dbc-be97-6c05b5efced2-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:08:10.153380 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:10.153347    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"
Nov 03 01:08:10.206882 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:10.206837    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"]
Nov 03 01:08:10.218419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:10.218382    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-537652-6c68d697c9-hnq46"]
Nov 03 01:08:10.570306 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:10.570121    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28473950-3c2f-4dbc-be97-6c05b5efced2" path="/var/lib/kubelet/pods/28473950-3c2f-4dbc-be97-6c05b5efced2/volumes"
Nov 03 01:08:17.578823 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:17.578789    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:08:18.107806 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:18.107776    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:08:18.108034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:18.108013    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:08:19.907893 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:08:19.907854    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:08:31.718230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.715452    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"]
Nov 03 01:08:31.718230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.716423    6000 cpu_manager.go:414] "containerMap: removing container" podUID="28473950-3c2f-4dbc-be97-6c05b5efced2" containerName="maas-api"
Nov 03 01:08:31.718230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.716441    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28473950-3c2f-4dbc-be97-6c05b5efced2" containerName="maas-api"
Nov 03 01:08:31.718230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.716555    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="28473950-3c2f-4dbc-be97-6c05b5efced2" containerName="maas-api"
Nov 03 01:08:31.722985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.722124    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.740061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.740036    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"]
Nov 03 01:08:31.748299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.746325    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"kube-root-ca.crt\""
Nov 03 01:08:31.748299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.747665    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"openshift-service-ca.crt\""
Nov 03 01:08:31.748299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.747754    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"rate-test-model-065de0-kserve-self-signed-certs\""
Nov 03 01:08:31.748299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.747797    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-065de0\"/\"default-dockercfg-897q9\""
Nov 03 01:08:31.874715 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.874677    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6xhh\" (UniqueName: \"kubernetes.io/projected/5c972578-bd59-4b6b-a4fa-686f8c44d97b-kube-api-access-n6xhh\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.874931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.874909    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-dshm\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.875002 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.874965    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-model-cache\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.875114 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.875092    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tls-certs\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.875235 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.875217    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tmp-dir\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.875362 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.875346    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-home\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.979327    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-dshm\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.979382    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-model-cache\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.979422    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tls-certs\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.979507    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tmp-dir\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.979587    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-home\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.979650    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6xhh\" (UniqueName: \"kubernetes.io/projected/5c972578-bd59-4b6b-a4fa-686f8c44d97b-kube-api-access-n6xhh\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.984629    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tmp-dir\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.984909    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-model-cache\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.985489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.985136    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-home\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:31.999571 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:31.999543    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-dshm\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:32.022620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:32.022581    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tls-certs\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:32.024673 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:32.024644    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6xhh\" (UniqueName: \"kubernetes.io/projected/5c972578-bd59-4b6b-a4fa-686f8c44d97b-kube-api-access-n6xhh\") pod \"rate-test-model-065de0-kserve-964578485-p6nnm\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") " pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:32.039857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:32.039584    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:32.358850 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:08:32.358821    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5c972578_bd59_4b6b_a4fa_686f8c44d97b.slice/crio-89f907d7ec3b1bd8ca10441e8b373d26ee9e6c56855465677c20cc381fe8aaf7 WatchSource:0}: Error finding container 89f907d7ec3b1bd8ca10441e8b373d26ee9e6c56855465677c20cc381fe8aaf7: Status 404 returned error can't find the container with id 89f907d7ec3b1bd8ca10441e8b373d26ee9e6c56855465677c20cc381fe8aaf7
Nov 03 01:08:32.359394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:32.359326    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"]
Nov 03 01:08:33.350454 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:33.350390    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm" event={"ID":"5c972578-bd59-4b6b-a4fa-686f8c44d97b","Type":"ContainerStarted","Data":"41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144"}
Nov 03 01:08:33.350933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:33.350516    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm" event={"ID":"5c972578-bd59-4b6b-a4fa-686f8c44d97b","Type":"ContainerStarted","Data":"89f907d7ec3b1bd8ca10441e8b373d26ee9e6c56855465677c20cc381fe8aaf7"}
Nov 03 01:08:33.389917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:33.389863    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm" podStartSLOduration=2.389846902 podStartE2EDuration="2.389846902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:08:33.387763053 +0000 UTC m=+3750.943115077" watchObservedRunningTime="2026-11-03 01:08:33.389846902 +0000 UTC m=+3750.945198943"
Nov 03 01:08:34.040664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:34.040613    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:42.040779 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:42.040734    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:42.053648 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:42.053621    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:42.621363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:42.621335    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:08:49.897807 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:08:49.897773    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:08:59.910369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:08:59.910330    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:09:06.520986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:06.520956    6000 scope.go:122] "RemoveContainer" containerID="15cbf560f5227e06d19cc8bdad49771dd2f787df82e109ec1cec877a345aa30b"
Nov 03 01:09:14.234506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:14.234421    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"]
Nov 03 01:09:14.265518 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:14.265490    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"]
Nov 03 01:09:19.235790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:19.235745    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" podUID="e8c90679-b97d-4691-9fa4-352d916678dd" containerName="payload-processing" containerID="cri-o://c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6" gracePeriod=25
Nov 03 01:09:19.266169 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:19.266130    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" podUID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" containerName="payload-pre-processing" containerID="cri-o://ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b" gracePeriod=25
Nov 03 01:09:19.897536 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:09:19.897494    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:09:26.277278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:26.276319    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" podUID="e8c90679-b97d-4691-9fa4-352d916678dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:09:28.324723 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:28.324680    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" podUID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:09:36.276176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:36.276139    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" podUID="e8c90679-b97d-4691-9fa4-352d916678dd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:09:38.323840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:38.323801    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" podUID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:09:39.549223 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.549193    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 01:09:39.623550 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.623520    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8c90679-b97d-4691-9fa4-352d916678dd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c90679-b97d-4691-9fa4-352d916678dd-plugins-config-volume\") pod \"e8c90679-b97d-4691-9fa4-352d916678dd\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") "
Nov 03 01:09:39.624034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.623826    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8c90679-b97d-4691-9fa4-352d916678dd-kube-api-access-462n9\" (UniqueName: \"kubernetes.io/projected/e8c90679-b97d-4691-9fa4-352d916678dd-kube-api-access-462n9\") pod \"e8c90679-b97d-4691-9fa4-352d916678dd\" (UID: \"e8c90679-b97d-4691-9fa4-352d916678dd\") "
Nov 03 01:09:39.624034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.623867    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8c90679-b97d-4691-9fa4-352d916678dd-plugins-config-volume" pod "e8c90679-b97d-4691-9fa4-352d916678dd" (UID: "e8c90679-b97d-4691-9fa4-352d916678dd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:09:39.624899 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.624874    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e8c90679-b97d-4691-9fa4-352d916678dd-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:09:39.626516 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.626490    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8c90679-b97d-4691-9fa4-352d916678dd-kube-api-access-462n9" pod "e8c90679-b97d-4691-9fa4-352d916678dd" (UID: "e8c90679-b97d-4691-9fa4-352d916678dd"). InnerVolumeSpecName "kube-api-access-462n9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:39.629472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.629449    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 01:09:39.725739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.725711    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-kube-api-access-wprmn\" (UniqueName: \"kubernetes.io/projected/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-kube-api-access-wprmn\") pod \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") "
Nov 03 01:09:39.725931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.725880    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-plugins-config-volume\") pod \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\" (UID: \"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b\") "
Nov 03 01:09:39.726236 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.726218    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-462n9\" (UniqueName: \"kubernetes.io/projected/e8c90679-b97d-4691-9fa4-352d916678dd-kube-api-access-462n9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:09:39.726433 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.726235    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-plugins-config-volume" pod "0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" (UID: "0d2f8092-9d61-4b06-a9d4-a304c85d9f8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:09:39.728055 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.728029    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-kube-api-access-wprmn" pod "0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" (UID: "0d2f8092-9d61-4b06-a9d4-a304c85d9f8b"). InnerVolumeSpecName "kube-api-access-wprmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:09:39.828134 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.828055    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:09:39.828134 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:39.828086    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wprmn\" (UniqueName: \"kubernetes.io/projected/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b-kube-api-access-wprmn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:09:40.137215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.137179    6000 generic.go:356] "Generic (PLEG): container finished" podID="e8c90679-b97d-4691-9fa4-352d916678dd" containerID="c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6" exitCode=0
Nov 03 01:09:40.137480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.137229    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"
Nov 03 01:09:40.137480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.137282    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" event={"ID":"e8c90679-b97d-4691-9fa4-352d916678dd","Type":"ContainerDied","Data":"c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6"}
Nov 03 01:09:40.137480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.137343    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z" event={"ID":"e8c90679-b97d-4691-9fa4-352d916678dd","Type":"ContainerDied","Data":"e86b418ee03ff982fcb08350c0dcab01dc18f017b11fe246d6cf4a23731b362d"}
Nov 03 01:09:40.137480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.137376    6000 scope.go:122] "RemoveContainer" containerID="c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6"
Nov 03 01:09:40.138807 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.138782    6000 generic.go:356] "Generic (PLEG): container finished" podID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" containerID="ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b" exitCode=0
Nov 03 01:09:40.138950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.138832    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"
Nov 03 01:09:40.138950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.138863    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" event={"ID":"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b","Type":"ContainerDied","Data":"ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b"}
Nov 03 01:09:40.138950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.138920    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6" event={"ID":"0d2f8092-9d61-4b06-a9d4-a304c85d9f8b","Type":"ContainerDied","Data":"4bface825c4b62b89fc7168235630a996b17e1194ade1b08a33c32180a9bfeb7"}
Nov 03 01:09:40.149294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.149279    6000 scope.go:122] "RemoveContainer" containerID="c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6"
Nov 03 01:09:40.149612 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:09:40.149594    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6\": container with ID starting with c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6 not found: ID does not exist" containerID="c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6"
Nov 03 01:09:40.149713 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.149623    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6"} err="failed to get container status \"c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6\": rpc error: code = NotFound desc = could not find container \"c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6\": container with ID starting with c018084e42c2d6826a698cff352b0d1720ee3f2f9adc41c42288b5b0c32e7df6 not found: ID does not exist"
Nov 03 01:09:40.149713 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.149644    6000 scope.go:122] "RemoveContainer" containerID="ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b"
Nov 03 01:09:40.159654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.159636    6000 scope.go:122] "RemoveContainer" containerID="ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b"
Nov 03 01:09:40.159928 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:09:40.159906    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b\": container with ID starting with ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b not found: ID does not exist" containerID="ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b"
Nov 03 01:09:40.159979 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.159938    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b"} err="failed to get container status \"ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b\": rpc error: code = NotFound desc = could not find container \"ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b\": container with ID starting with ff0b3140410bf54979e170b62aaa2022438b73cbb3904b3f96abf462d357c58b not found: ID does not exist"
Nov 03 01:09:40.210280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.209336    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"]
Nov 03 01:09:40.225159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.225120    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-d87781-68568f5fc-ckk2z"]
Nov 03 01:09:40.241360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.241324    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"]
Nov 03 01:09:40.245374 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.245346    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-d87781-5574866475-bskp6"]
Nov 03 01:09:40.568578 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.568499    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" path="/var/lib/kubelet/pods/0d2f8092-9d61-4b06-a9d4-a304c85d9f8b/volumes"
Nov 03 01:09:40.568959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:09:40.568926    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8c90679-b97d-4691-9fa4-352d916678dd" path="/var/lib/kubelet/pods/e8c90679-b97d-4691-9fa4-352d916678dd/volumes"
Nov 03 01:09:49.898415 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:09:49.898377    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:10:03.507035 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:03.506997    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"]
Nov 03 01:10:03.507488 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:03.507283    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy" containerID="cri-o://7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5" gracePeriod=30
Nov 03 01:10:08.826797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.826768    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 01:10:08.964097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964059    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-data\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964338 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964118    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-kube-api-access-8l8fh\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-kube-api-access-8l8fh\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964338 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964156    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-credential-socket\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964338 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964174    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-envoy\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964560 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964366    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istiod-ca-cert\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964560 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964391    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-data" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:08.964560 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964419    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-certs\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964560 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964464    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-credential-socket" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:08.964786 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964564    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-token\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-token\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964786 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964675    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-certs" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:08.964786 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964723    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-podinfo\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964786 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964758    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-socket\") pod \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\" (UID: \"587e6577-0386-4e2c-85e2-8e91b40c5fbc\") "
Nov 03 01:10:08.964786 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.964770    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istiod-ca-cert" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:08.965392 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.965224    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-socket" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:08.965727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.965703    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:08.965826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.965733    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:08.965826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.965751    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:08.965826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.965765    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:08.965826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.965779    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:08.966892 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.966858    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-kube-api-access-8l8fh" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "kube-api-access-8l8fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:08.967006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.966940    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-token" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:08.967072 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.967035    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-envoy" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:10:08.967175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:08.967159    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-podinfo" pod "587e6577-0386-4e2c-85e2-8e91b40c5fbc" (UID: "587e6577-0386-4e2c-85e2-8e91b40c5fbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:10:09.066980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.066900    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:09.066980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.066929    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:09.066980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.066940    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l8fh\" (UniqueName: \"kubernetes.io/projected/587e6577-0386-4e2c-85e2-8e91b40c5fbc-kube-api-access-8l8fh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:09.066980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.066950    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/587e6577-0386-4e2c-85e2-8e91b40c5fbc-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:09.198389 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.198358    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:10:09.426436 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.426407    6000 generic.go:356] "Generic (PLEG): container finished" podID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerID="7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5" exitCode=0
Nov 03 01:10:09.426609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.426442    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" event={"ID":"587e6577-0386-4e2c-85e2-8e91b40c5fbc","Type":"ContainerDied","Data":"7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5"}
Nov 03 01:10:09.426609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.426462    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"
Nov 03 01:10:09.426609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.426501    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp" event={"ID":"587e6577-0386-4e2c-85e2-8e91b40c5fbc","Type":"ContainerDied","Data":"7f5ec2280a65857dd05fa37b63f347e12bf2812b303760653d2ac368d8021865"}
Nov 03 01:10:09.426609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.426527    6000 scope.go:122] "RemoveContainer" containerID="7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5"
Nov 03 01:10:09.437956 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.437939    6000 scope.go:122] "RemoveContainer" containerID="206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f"
Nov 03 01:10:09.447574 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.447559    6000 scope.go:122] "RemoveContainer" containerID="7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5"
Nov 03 01:10:09.447843 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:09.447822    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5\": container with ID starting with 7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5 not found: ID does not exist" containerID="7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5"
Nov 03 01:10:09.447952 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.447849    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5"} err="failed to get container status \"7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5\": rpc error: code = NotFound desc = could not find container \"7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5\": container with ID starting with 7765035c7ea37432cb1a2d9be3b7695ef72e477ea0f6728de835e7b4b44a16f5 not found: ID does not exist"
Nov 03 01:10:09.447952 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.447865    6000 scope.go:122] "RemoveContainer" containerID="206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f"
Nov 03 01:10:09.448080 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:09.448060    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f\": container with ID starting with 206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f not found: ID does not exist" containerID="206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f"
Nov 03 01:10:09.448121 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.448087    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f"} err="failed to get container status \"206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f\": rpc error: code = NotFound desc = could not find container \"206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f\": container with ID starting with 206604c1212960d3817f6067255f0b9af42b0b92371f972256b70fc2bbee644f not found: ID does not exist"
Nov 03 01:10:09.479514 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.479447    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"]
Nov 03 01:10:09.490799 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:09.490776    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-d87781-openshift-default-57f799fb65-dlcgp"]
Nov 03 01:10:10.569589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:10.569553    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" path="/var/lib/kubelet/pods/587e6577-0386-4e2c-85e2-8e91b40c5fbc/volumes"
Nov 03 01:10:19.898484 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:19.898441    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:10:22.881824 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:22.881785    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"]
Nov 03 01:10:22.882608 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:22.882085    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh" podUID="4729ad2b-ba3f-47f0-9cae-7c843a10781f" containerName="maas-api" containerID="cri-o://ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1" gracePeriod=30
Nov 03 01:10:23.169991 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.169963    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 01:10:23.280649 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.280617    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-maas-api-tls\") pod \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") "
Nov 03 01:10:23.280857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.280752    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4729ad2b-ba3f-47f0-9cae-7c843a10781f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729ad2b-ba3f-47f0-9cae-7c843a10781f-openshift-service-ca\") pod \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") "
Nov 03 01:10:23.280857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.280832    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-metrics-certs\") pod \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") "
Nov 03 01:10:23.280957 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.280890    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4729ad2b-ba3f-47f0-9cae-7c843a10781f-kube-api-access-f8sv6\" (UniqueName: \"kubernetes.io/projected/4729ad2b-ba3f-47f0-9cae-7c843a10781f-kube-api-access-f8sv6\") pod \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\" (UID: \"4729ad2b-ba3f-47f0-9cae-7c843a10781f\") "
Nov 03 01:10:23.281175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.281149    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4729ad2b-ba3f-47f0-9cae-7c843a10781f-openshift-service-ca" pod "4729ad2b-ba3f-47f0-9cae-7c843a10781f" (UID: "4729ad2b-ba3f-47f0-9cae-7c843a10781f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:23.281821 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.281764    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4729ad2b-ba3f-47f0-9cae-7c843a10781f-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.283188 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.283150    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-metrics-certs" pod "4729ad2b-ba3f-47f0-9cae-7c843a10781f" (UID: "4729ad2b-ba3f-47f0-9cae-7c843a10781f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:10:23.283382 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.283364    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-maas-api-tls" pod "4729ad2b-ba3f-47f0-9cae-7c843a10781f" (UID: "4729ad2b-ba3f-47f0-9cae-7c843a10781f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:10:23.283447 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.283409    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4729ad2b-ba3f-47f0-9cae-7c843a10781f-kube-api-access-f8sv6" pod "4729ad2b-ba3f-47f0-9cae-7c843a10781f" (UID: "4729ad2b-ba3f-47f0-9cae-7c843a10781f"). InnerVolumeSpecName "kube-api-access-f8sv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:23.399494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.396014    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.399494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.396053    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8sv6\" (UniqueName: \"kubernetes.io/projected/4729ad2b-ba3f-47f0-9cae-7c843a10781f-kube-api-access-f8sv6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.399494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.396167    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4729ad2b-ba3f-47f0-9cae-7c843a10781f-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:23.543412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.543380    6000 generic.go:356] "Generic (PLEG): container finished" podID="4729ad2b-ba3f-47f0-9cae-7c843a10781f" containerID="ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1" exitCode=0
Nov 03 01:10:23.543598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.543427    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"
Nov 03 01:10:23.543598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.543462    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh" event={"ID":"4729ad2b-ba3f-47f0-9cae-7c843a10781f","Type":"ContainerDied","Data":"ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1"}
Nov 03 01:10:23.543598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.543519    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh" event={"ID":"4729ad2b-ba3f-47f0-9cae-7c843a10781f","Type":"ContainerDied","Data":"fb8739c5378dc85168208a4a97f9d224a31f3a097a2b5dc402fdd8a125f5647d"}
Nov 03 01:10:23.543598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.543550    6000 scope.go:122] "RemoveContainer" containerID="ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1"
Nov 03 01:10:23.555070 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.555050    6000 scope.go:122] "RemoveContainer" containerID="ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1"
Nov 03 01:10:23.555370 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:23.555350    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1\": container with ID starting with ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1 not found: ID does not exist" containerID="ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1"
Nov 03 01:10:23.555449 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.555377    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1"} err="failed to get container status \"ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1\": rpc error: code = NotFound desc = could not find container \"ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1\": container with ID starting with ff5e39ee633ac7318509cd7484bec979a34b8d0e291673ffbc326e9463d88ae1 not found: ID does not exist"
Nov 03 01:10:23.596790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.596757    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"]
Nov 03 01:10:23.607868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:23.607838    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-955dc3-5d7875d5f9-k4fbh"]
Nov 03 01:10:24.568767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:24.568692    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4729ad2b-ba3f-47f0-9cae-7c843a10781f" path="/var/lib/kubelet/pods/4729ad2b-ba3f-47f0-9cae-7c843a10781f/volumes"
Nov 03 01:10:26.119732 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:26.119691    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"]
Nov 03 01:10:26.138363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:26.138333    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"]
Nov 03 01:10:31.120125 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:31.120079    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" podUID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" containerName="payload-processing" containerID="cri-o://81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e" gracePeriod=25
Nov 03 01:10:31.138867 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:31.138799    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" podUID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" containerName="payload-pre-processing" containerID="cri-o://55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6" gracePeriod=25
Nov 03 01:10:31.770088 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:31.770047    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" podUID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:31.770307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:31.770058    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" podUID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:33.953468 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:33.953425    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"]
Nov 03 01:10:33.953993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:33.953651    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg" podUID="40b24645-1feb-4ed5-b735-cb817f512708" containerName="maas-api" containerID="cri-o://6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660" gracePeriod=30
Nov 03 01:10:34.302175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.302148    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 01:10:34.388676 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.388532    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40b24645-1feb-4ed5-b735-cb817f512708-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b24645-1feb-4ed5-b735-cb817f512708-openshift-service-ca\") pod \"40b24645-1feb-4ed5-b735-cb817f512708\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") "
Nov 03 01:10:34.389628 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.389601    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-metrics-certs\") pod \"40b24645-1feb-4ed5-b735-cb817f512708\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") "
Nov 03 01:10:34.389817 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.389800    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-maas-api-tls\") pod \"40b24645-1feb-4ed5-b735-cb817f512708\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") "
Nov 03 01:10:34.390079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.390063    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40b24645-1feb-4ed5-b735-cb817f512708-kube-api-access-wczrj\" (UniqueName: \"kubernetes.io/projected/40b24645-1feb-4ed5-b735-cb817f512708-kube-api-access-wczrj\") pod \"40b24645-1feb-4ed5-b735-cb817f512708\" (UID: \"40b24645-1feb-4ed5-b735-cb817f512708\") "
Nov 03 01:10:34.391376 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.389415    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40b24645-1feb-4ed5-b735-cb817f512708-openshift-service-ca" pod "40b24645-1feb-4ed5-b735-cb817f512708" (UID: "40b24645-1feb-4ed5-b735-cb817f512708"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:34.448774 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.448713    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40b24645-1feb-4ed5-b735-cb817f512708-kube-api-access-wczrj" pod "40b24645-1feb-4ed5-b735-cb817f512708" (UID: "40b24645-1feb-4ed5-b735-cb817f512708"). InnerVolumeSpecName "kube-api-access-wczrj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:34.469998 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.456396    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-metrics-certs" pod "40b24645-1feb-4ed5-b735-cb817f512708" (UID: "40b24645-1feb-4ed5-b735-cb817f512708"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:10:34.469998 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.456491    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-maas-api-tls" pod "40b24645-1feb-4ed5-b735-cb817f512708" (UID: "40b24645-1feb-4ed5-b735-cb817f512708"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:10:34.498148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.497035    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:34.498148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.497078    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40b24645-1feb-4ed5-b735-cb817f512708-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:34.498148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.497093    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wczrj\" (UniqueName: \"kubernetes.io/projected/40b24645-1feb-4ed5-b735-cb817f512708-kube-api-access-wczrj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:34.498148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.497110    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40b24645-1feb-4ed5-b735-cb817f512708-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:34.656074 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.656037    6000 generic.go:356] "Generic (PLEG): container finished" podID="40b24645-1feb-4ed5-b735-cb817f512708" containerID="6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660" exitCode=0
Nov 03 01:10:34.656352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.656105    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"
Nov 03 01:10:34.656352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.656129    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg" event={"ID":"40b24645-1feb-4ed5-b735-cb817f512708","Type":"ContainerDied","Data":"6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660"}
Nov 03 01:10:34.656352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.656224    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg" event={"ID":"40b24645-1feb-4ed5-b735-cb817f512708","Type":"ContainerDied","Data":"bc3005eaf707cb6d3509a700011419919550213cabf22993ea40f6cf8e8ac720"}
Nov 03 01:10:34.656352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.656247    6000 scope.go:122] "RemoveContainer" containerID="6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660"
Nov 03 01:10:34.672056 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.672035    6000 scope.go:122] "RemoveContainer" containerID="6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660"
Nov 03 01:10:34.672392 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:34.672360    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660\": container with ID starting with 6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660 not found: ID does not exist" containerID="6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660"
Nov 03 01:10:34.672489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.672404    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660"} err="failed to get container status \"6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660\": rpc error: code = NotFound desc = could not find container \"6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660\": container with ID starting with 6821332dbdcdbd49e59fa04900c5bbd1eeccf797adc3b41e41ecfd2298159660 not found: ID does not exist"
Nov 03 01:10:34.706340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.706304    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"]
Nov 03 01:10:34.717581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:34.717556    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-359f71-848cbb4754-jxfpg"]
Nov 03 01:10:36.569451 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:36.569417    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40b24645-1feb-4ed5-b735-cb817f512708" path="/var/lib/kubelet/pods/40b24645-1feb-4ed5-b735-cb817f512708/volumes"
Nov 03 01:10:41.302610 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:41.302570    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"]
Nov 03 01:10:41.337865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:41.337824    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"]
Nov 03 01:10:41.769844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:41.769801    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" podUID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:41.770046 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:41.769800    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" podUID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:46.303048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:46.302992    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" podUID="78254f06-bf41-44d1-8c79-b236c12f380e" containerName="payload-processing" containerID="cri-o://ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e" gracePeriod=25
Nov 03 01:10:46.338625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:46.338593    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" podUID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" containerName="payload-pre-processing" containerID="cri-o://1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282" gracePeriod=25
Nov 03 01:10:47.555633 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:47.555589    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"]
Nov 03 01:10:47.588074 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:47.588039    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"]
Nov 03 01:10:49.897492 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:49.897457    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:10:51.419018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.418991    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 01:10:51.463495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.463470    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 01:10:51.546749 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.546718    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-kube-api-access-7dx24\" (UniqueName: \"kubernetes.io/projected/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-kube-api-access-7dx24\") pod \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") "
Nov 03 01:10:51.546959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.546853    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-plugins-config-volume\") pod \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") "
Nov 03 01:10:51.547049 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.546982    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-plugins-config-volume\") pod \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\" (UID: \"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372\") "
Nov 03 01:10:51.547124 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.547080    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-kube-api-access-ct9pp\" (UniqueName: \"kubernetes.io/projected/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-kube-api-access-ct9pp\") pod \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\" (UID: \"aff0a103-896a-43e3-a6b9-c02f8ffa23ff\") "
Nov 03 01:10:51.547313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.547284    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-plugins-config-volume" pod "aff0a103-896a-43e3-a6b9-c02f8ffa23ff" (UID: "aff0a103-896a-43e3-a6b9-c02f8ffa23ff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:51.547818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.547773    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:51.547818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.547782    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-plugins-config-volume" pod "7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" (UID: "7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:10:51.549350 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.549330    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-kube-api-access-7dx24" pod "7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" (UID: "7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372"). InnerVolumeSpecName "kube-api-access-7dx24". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:51.549800 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.549783    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-kube-api-access-ct9pp" pod "aff0a103-896a-43e3-a6b9-c02f8ffa23ff" (UID: "aff0a103-896a-43e3-a6b9-c02f8ffa23ff"). InnerVolumeSpecName "kube-api-access-ct9pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:10:51.649303 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.649193    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dx24\" (UniqueName: \"kubernetes.io/projected/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-kube-api-access-7dx24\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:51.649303 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.649225    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:51.649303 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.649237    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct9pp\" (UniqueName: \"kubernetes.io/projected/aff0a103-896a-43e3-a6b9-c02f8ffa23ff-kube-api-access-ct9pp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:10:51.871858 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.871819    6000 generic.go:356] "Generic (PLEG): container finished" podID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" containerID="55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6" exitCode=0
Nov 03 01:10:51.872069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.871867    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"
Nov 03 01:10:51.872069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.871901    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" event={"ID":"aff0a103-896a-43e3-a6b9-c02f8ffa23ff","Type":"ContainerDied","Data":"55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6"}
Nov 03 01:10:51.872069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.871966    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs" event={"ID":"aff0a103-896a-43e3-a6b9-c02f8ffa23ff","Type":"ContainerDied","Data":"fc3162907b090b285e037292c747ae4446810b56421f23734eeeae98612c0af4"}
Nov 03 01:10:51.872069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.871989    6000 scope.go:122] "RemoveContainer" containerID="55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6"
Nov 03 01:10:51.874014 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.873988    6000 generic.go:356] "Generic (PLEG): container finished" podID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" containerID="81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e" exitCode=0
Nov 03 01:10:51.874163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.874032    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"
Nov 03 01:10:51.874163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.874149    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" event={"ID":"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372","Type":"ContainerDied","Data":"81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e"}
Nov 03 01:10:51.874317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.874198    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf" event={"ID":"7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372","Type":"ContainerDied","Data":"e797724e78988f2b473fb92affc3a8c1b4e005b280cbd1aed3ea7759bfb7c066"}
Nov 03 01:10:51.883619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.883596    6000 scope.go:122] "RemoveContainer" containerID="55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6"
Nov 03 01:10:51.883954 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:51.883918    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6\": container with ID starting with 55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6 not found: ID does not exist" containerID="55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6"
Nov 03 01:10:51.884040 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.883968    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6"} err="failed to get container status \"55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6\": rpc error: code = NotFound desc = could not find container \"55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6\": container with ID starting with 55042e17a64128d6434d19d6366891f845af737576423a7303971d4ca66ffdd6 not found: ID does not exist"
Nov 03 01:10:51.884040 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.883993    6000 scope.go:122] "RemoveContainer" containerID="81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e"
Nov 03 01:10:51.893596 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.893580    6000 scope.go:122] "RemoveContainer" containerID="81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e"
Nov 03 01:10:51.893869 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:10:51.893850    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e\": container with ID starting with 81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e not found: ID does not exist" containerID="81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e"
Nov 03 01:10:51.893919 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.893876    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e"} err="failed to get container status \"81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e\": rpc error: code = NotFound desc = could not find container \"81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e\": container with ID starting with 81e3d2f00f60b648e99116419ea30bda721ce1102094da53f58ea4d287727f1e not found: ID does not exist"
Nov 03 01:10:51.922423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.922360    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"]
Nov 03 01:10:51.933543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.933522    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-359f71-754cff6775-smzzs"]
Nov 03 01:10:51.957214 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.957189    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"]
Nov 03 01:10:51.961753 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.961733    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-359f71-6976dfcb89-tw2nf"]
Nov 03 01:10:51.975718 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.975695    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" podUID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:51.975838 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:51.975692    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" podUID="78254f06-bf41-44d1-8c79-b236c12f380e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:52.097332 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:52.097298    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:10:52.556654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:52.556598    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" podUID="d17d7d40-68cf-482f-86fa-34f5af94813c" containerName="payload-processing" containerID="cri-o://72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28" gracePeriod=25
Nov 03 01:10:52.569164 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:52.569125    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" path="/var/lib/kubelet/pods/7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372/volumes"
Nov 03 01:10:52.569695 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:52.569678    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" path="/var/lib/kubelet/pods/aff0a103-896a-43e3-a6b9-c02f8ffa23ff/volumes"
Nov 03 01:10:52.588606 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:52.588575    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" podUID="2f71ee87-6e9f-4e44-b400-577011399180" containerName="payload-pre-processing" containerID="cri-o://3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a" gracePeriod=25
Nov 03 01:10:56.266913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:56.266868    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" podUID="d17d7d40-68cf-482f-86fa-34f5af94813c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:10:56.266913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:10:56.266902    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" podUID="2f71ee87-6e9f-4e44-b400-577011399180" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:01.976155 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:01.976114    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" podUID="78254f06-bf41-44d1-8c79-b236c12f380e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:01.976704 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:01.976169    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" podUID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:06.266930 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.266891    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" podUID="2f71ee87-6e9f-4e44-b400-577011399180" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:06.267479 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.266932    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" podUID="d17d7d40-68cf-482f-86fa-34f5af94813c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:06.642507 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.642471    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 01:11:06.670619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.670588    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 01:11:06.804007 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.803919    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7811f9d-d30d-4fee-850f-10c22d42e2cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7811f9d-d30d-4fee-850f-10c22d42e2cc-plugins-config-volume\") pod \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") "
Nov 03 01:11:06.804007 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.803959    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7811f9d-d30d-4fee-850f-10c22d42e2cc-kube-api-access-xcjbj\" (UniqueName: \"kubernetes.io/projected/a7811f9d-d30d-4fee-850f-10c22d42e2cc-kube-api-access-xcjbj\") pod \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\" (UID: \"a7811f9d-d30d-4fee-850f-10c22d42e2cc\") "
Nov 03 01:11:06.804201 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.804026    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78254f06-bf41-44d1-8c79-b236c12f380e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78254f06-bf41-44d1-8c79-b236c12f380e-plugins-config-volume\") pod \"78254f06-bf41-44d1-8c79-b236c12f380e\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") "
Nov 03 01:11:06.804351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.804330    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78254f06-bf41-44d1-8c79-b236c12f380e-kube-api-access-48vvl\" (UniqueName: \"kubernetes.io/projected/78254f06-bf41-44d1-8c79-b236c12f380e-kube-api-access-48vvl\") pod \"78254f06-bf41-44d1-8c79-b236c12f380e\" (UID: \"78254f06-bf41-44d1-8c79-b236c12f380e\") "
Nov 03 01:11:06.804441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.804420    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7811f9d-d30d-4fee-850f-10c22d42e2cc-plugins-config-volume" pod "a7811f9d-d30d-4fee-850f-10c22d42e2cc" (UID: "a7811f9d-d30d-4fee-850f-10c22d42e2cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:06.804510 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.804432    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78254f06-bf41-44d1-8c79-b236c12f380e-plugins-config-volume" pod "78254f06-bf41-44d1-8c79-b236c12f380e" (UID: "78254f06-bf41-44d1-8c79-b236c12f380e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:06.804758 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.804741    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a7811f9d-d30d-4fee-850f-10c22d42e2cc-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:06.804854 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.804763    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/78254f06-bf41-44d1-8c79-b236c12f380e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:06.806377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.806355    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7811f9d-d30d-4fee-850f-10c22d42e2cc-kube-api-access-xcjbj" pod "a7811f9d-d30d-4fee-850f-10c22d42e2cc" (UID: "a7811f9d-d30d-4fee-850f-10c22d42e2cc"). InnerVolumeSpecName "kube-api-access-xcjbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:06.806539 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.806517    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78254f06-bf41-44d1-8c79-b236c12f380e-kube-api-access-48vvl" pod "78254f06-bf41-44d1-8c79-b236c12f380e" (UID: "78254f06-bf41-44d1-8c79-b236c12f380e"). InnerVolumeSpecName "kube-api-access-48vvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:06.906237 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.906207    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48vvl\" (UniqueName: \"kubernetes.io/projected/78254f06-bf41-44d1-8c79-b236c12f380e-kube-api-access-48vvl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:06.906237 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.906235    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcjbj\" (UniqueName: \"kubernetes.io/projected/a7811f9d-d30d-4fee-850f-10c22d42e2cc-kube-api-access-xcjbj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:06.989483 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.989450    6000 generic.go:356] "Generic (PLEG): container finished" podID="78254f06-bf41-44d1-8c79-b236c12f380e" containerID="ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e" exitCode=0
Nov 03 01:11:06.989710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.989507    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"
Nov 03 01:11:06.989710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.989541    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" event={"ID":"78254f06-bf41-44d1-8c79-b236c12f380e","Type":"ContainerDied","Data":"ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e"}
Nov 03 01:11:06.989710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.989595    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk" event={"ID":"78254f06-bf41-44d1-8c79-b236c12f380e","Type":"ContainerDied","Data":"b8e30a724bf8af57733e22ae198c5872d0ed5d20db440308ced9c5a31a254bc7"}
Nov 03 01:11:06.989710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.989627    6000 scope.go:122] "RemoveContainer" containerID="ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e"
Nov 03 01:11:06.991085 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.991059    6000 generic.go:356] "Generic (PLEG): container finished" podID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" containerID="1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282" exitCode=0
Nov 03 01:11:06.991419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.991092    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" event={"ID":"a7811f9d-d30d-4fee-850f-10c22d42e2cc","Type":"ContainerDied","Data":"1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282"}
Nov 03 01:11:06.991419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.991119    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"
Nov 03 01:11:06.991419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:06.991131    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg" event={"ID":"a7811f9d-d30d-4fee-850f-10c22d42e2cc","Type":"ContainerDied","Data":"7fee264a90c52948ede5a4c9e6ef92a37ce431af935b94dd878b88c7df50d9de"}
Nov 03 01:11:07.001686 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.001662    6000 scope.go:122] "RemoveContainer" containerID="ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e"
Nov 03 01:11:07.003787 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:07.003760    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e\": container with ID starting with ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e not found: ID does not exist" containerID="ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e"
Nov 03 01:11:07.003913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.003792    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e"} err="failed to get container status \"ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e\": rpc error: code = NotFound desc = could not find container \"ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e\": container with ID starting with ac07a72756da240af5d40e223ec2225c5ed3b84f10a0e6f4692ebc4d36099f7e not found: ID does not exist"
Nov 03 01:11:07.003913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.003812    6000 scope.go:122] "RemoveContainer" containerID="1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282"
Nov 03 01:11:07.014236 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.014218    6000 scope.go:122] "RemoveContainer" containerID="1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282"
Nov 03 01:11:07.014501 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:07.014481    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282\": container with ID starting with 1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282 not found: ID does not exist" containerID="1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282"
Nov 03 01:11:07.014548 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.014507    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282"} err="failed to get container status \"1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282\": rpc error: code = NotFound desc = could not find container \"1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282\": container with ID starting with 1c1fbdee65e1b2a413baa94c2ce10720953c0cee02c7c078c1122c6b26108282 not found: ID does not exist"
Nov 03 01:11:07.052068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.052041    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"]
Nov 03 01:11:07.057620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.057561    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-909268-6f94f47577-cv6fk"]
Nov 03 01:11:07.088791 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.088764    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"]
Nov 03 01:11:07.094804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:07.094782    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-909268-5ccf8444c9-49wkg"]
Nov 03 01:11:08.568369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:08.568339    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78254f06-bf41-44d1-8c79-b236c12f380e" path="/var/lib/kubelet/pods/78254f06-bf41-44d1-8c79-b236c12f380e/volumes"
Nov 03 01:11:08.568819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:08.568805    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" path="/var/lib/kubelet/pods/a7811f9d-d30d-4fee-850f-10c22d42e2cc/volumes"
Nov 03 01:11:12.934494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.934467    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 01:11:12.938751 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.938729    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 01:11:12.991434 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.991401    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f71ee87-6e9f-4e44-b400-577011399180-kube-api-access-w7qpj\" (UniqueName: \"kubernetes.io/projected/2f71ee87-6e9f-4e44-b400-577011399180-kube-api-access-w7qpj\") pod \"2f71ee87-6e9f-4e44-b400-577011399180\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") "
Nov 03 01:11:12.991641 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.991591    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17d7d40-68cf-482f-86fa-34f5af94813c-kube-api-access-2cs2x\" (UniqueName: \"kubernetes.io/projected/d17d7d40-68cf-482f-86fa-34f5af94813c-kube-api-access-2cs2x\") pod \"d17d7d40-68cf-482f-86fa-34f5af94813c\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") "
Nov 03 01:11:12.991641 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.991635    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f71ee87-6e9f-4e44-b400-577011399180-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f71ee87-6e9f-4e44-b400-577011399180-plugins-config-volume\") pod \"2f71ee87-6e9f-4e44-b400-577011399180\" (UID: \"2f71ee87-6e9f-4e44-b400-577011399180\") "
Nov 03 01:11:12.991787 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.991745    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17d7d40-68cf-482f-86fa-34f5af94813c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d7d40-68cf-482f-86fa-34f5af94813c-plugins-config-volume\") pod \"d17d7d40-68cf-482f-86fa-34f5af94813c\" (UID: \"d17d7d40-68cf-482f-86fa-34f5af94813c\") "
Nov 03 01:11:12.992185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.992119    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f71ee87-6e9f-4e44-b400-577011399180-plugins-config-volume" pod "2f71ee87-6e9f-4e44-b400-577011399180" (UID: "2f71ee87-6e9f-4e44-b400-577011399180"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:12.992472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.992242    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17d7d40-68cf-482f-86fa-34f5af94813c-plugins-config-volume" pod "d17d7d40-68cf-482f-86fa-34f5af94813c" (UID: "d17d7d40-68cf-482f-86fa-34f5af94813c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:12.992620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.992533    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f71ee87-6e9f-4e44-b400-577011399180-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:12.992620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.992562    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d17d7d40-68cf-482f-86fa-34f5af94813c-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:12.993963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.993931    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f71ee87-6e9f-4e44-b400-577011399180-kube-api-access-w7qpj" pod "2f71ee87-6e9f-4e44-b400-577011399180" (UID: "2f71ee87-6e9f-4e44-b400-577011399180"). InnerVolumeSpecName "kube-api-access-w7qpj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:12.994067 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:12.993961    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17d7d40-68cf-482f-86fa-34f5af94813c-kube-api-access-2cs2x" pod "d17d7d40-68cf-482f-86fa-34f5af94813c" (UID: "d17d7d40-68cf-482f-86fa-34f5af94813c"). InnerVolumeSpecName "kube-api-access-2cs2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:13.036977 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.036935    6000 generic.go:356] "Generic (PLEG): container finished" podID="2f71ee87-6e9f-4e44-b400-577011399180" containerID="3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a" exitCode=0
Nov 03 01:11:13.037159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.037007    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"
Nov 03 01:11:13.037159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.037030    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" event={"ID":"2f71ee87-6e9f-4e44-b400-577011399180","Type":"ContainerDied","Data":"3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a"}
Nov 03 01:11:13.037159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.037116    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl" event={"ID":"2f71ee87-6e9f-4e44-b400-577011399180","Type":"ContainerDied","Data":"16adf825e05e3bf4a1a753cdfb52107ec9f0dbc1e647b8ec0f753ed6209d007a"}
Nov 03 01:11:13.037402 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.037179    6000 scope.go:122] "RemoveContainer" containerID="3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a"
Nov 03 01:11:13.039782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.039555    6000 generic.go:356] "Generic (PLEG): container finished" podID="d17d7d40-68cf-482f-86fa-34f5af94813c" containerID="72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28" exitCode=0
Nov 03 01:11:13.039782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.039627    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"
Nov 03 01:11:13.039782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.039685    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" event={"ID":"d17d7d40-68cf-482f-86fa-34f5af94813c","Type":"ContainerDied","Data":"72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28"}
Nov 03 01:11:13.039782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.039728    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd" event={"ID":"d17d7d40-68cf-482f-86fa-34f5af94813c","Type":"ContainerDied","Data":"a0d160e99e437942ef0151047ef122fbcf3cbb3e35a4e98efe0234ccfb8fa6a4"}
Nov 03 01:11:13.050642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.050444    6000 scope.go:122] "RemoveContainer" containerID="3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a"
Nov 03 01:11:13.050806 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:13.050787    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a\": container with ID starting with 3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a not found: ID does not exist" containerID="3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a"
Nov 03 01:11:13.050859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.050817    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a"} err="failed to get container status \"3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a\": rpc error: code = NotFound desc = could not find container \"3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a\": container with ID starting with 3e0563c02e1811b3791e9202b5a686d7cb8d4b1cc9197bf97d6e972244b57d2a not found: ID does not exist"
Nov 03 01:11:13.050859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.050833    6000 scope.go:122] "RemoveContainer" containerID="72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28"
Nov 03 01:11:13.062192 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.062165    6000 scope.go:122] "RemoveContainer" containerID="72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28"
Nov 03 01:11:13.062535 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:13.062514    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28\": container with ID starting with 72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28 not found: ID does not exist" containerID="72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28"
Nov 03 01:11:13.062611 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.062544    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28"} err="failed to get container status \"72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28\": rpc error: code = NotFound desc = could not find container \"72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28\": container with ID starting with 72f2a75b7ba55594d4b6b14b52fba8ccd7c199377eb3f0e5f74dbbc303cb9f28 not found: ID does not exist"
Nov 03 01:11:13.086874 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.086846    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"]
Nov 03 01:11:13.089513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.089489    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-2858fb-84ff44d7c9-5vrkl"]
Nov 03 01:11:13.094174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.094109    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qpj\" (UniqueName: \"kubernetes.io/projected/2f71ee87-6e9f-4e44-b400-577011399180-kube-api-access-w7qpj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:13.094174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.094131    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cs2x\" (UniqueName: \"kubernetes.io/projected/d17d7d40-68cf-482f-86fa-34f5af94813c-kube-api-access-2cs2x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:13.134648 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.134602    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"]
Nov 03 01:11:13.140203 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:13.140179    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-2858fb-6749b44d75-mjvsd"]
Nov 03 01:11:14.602278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:14.589935    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f71ee87-6e9f-4e44-b400-577011399180" path="/var/lib/kubelet/pods/2f71ee87-6e9f-4e44-b400-577011399180/volumes"
Nov 03 01:11:14.602278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:14.591235    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17d7d40-68cf-482f-86fa-34f5af94813c" path="/var/lib/kubelet/pods/d17d7d40-68cf-482f-86fa-34f5af94813c/volumes"
Nov 03 01:11:18.596472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:18.596440    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"]
Nov 03 01:11:18.628936 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:18.628897    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"]
Nov 03 01:11:19.896715 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:19.896676    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:11:20.975727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:20.975687    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"]
Nov 03 01:11:23.596899 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.596863    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk" podUID="30a5aa39-82cd-479f-a09b-cab7db69e4ae" containerName="payload-processing" containerID="cri-o://8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f" gracePeriod=25
Nov 03 01:11:23.630096 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.630064    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl" podUID="69a516d6-a843-40be-9449-53a5cb69376d" containerName="payload-pre-processing" containerID="cri-o://e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364" gracePeriod=25
Nov 03 01:11:23.919883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.919850    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:11:23.962398 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.962373    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:11:23.990224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.990191    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30a5aa39-82cd-479f-a09b-cab7db69e4ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5aa39-82cd-479f-a09b-cab7db69e4ae-plugins-config-volume\") pod \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") "
Nov 03 01:11:23.990414 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.990278    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a516d6-a843-40be-9449-53a5cb69376d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a516d6-a843-40be-9449-53a5cb69376d-plugins-config-volume\") pod \"69a516d6-a843-40be-9449-53a5cb69376d\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") "
Nov 03 01:11:23.990414 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.990315    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a516d6-a843-40be-9449-53a5cb69376d-kube-api-access-6jbbx\" (UniqueName: \"kubernetes.io/projected/69a516d6-a843-40be-9449-53a5cb69376d-kube-api-access-6jbbx\") pod \"69a516d6-a843-40be-9449-53a5cb69376d\" (UID: \"69a516d6-a843-40be-9449-53a5cb69376d\") "
Nov 03 01:11:23.990414 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.990361    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30a5aa39-82cd-479f-a09b-cab7db69e4ae-kube-api-access-429z6\" (UniqueName: \"kubernetes.io/projected/30a5aa39-82cd-479f-a09b-cab7db69e4ae-kube-api-access-429z6\") pod \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\" (UID: \"30a5aa39-82cd-479f-a09b-cab7db69e4ae\") "
Nov 03 01:11:23.990756 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.990710    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30a5aa39-82cd-479f-a09b-cab7db69e4ae-plugins-config-volume" pod "30a5aa39-82cd-479f-a09b-cab7db69e4ae" (UID: "30a5aa39-82cd-479f-a09b-cab7db69e4ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:23.990924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.990774    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a516d6-a843-40be-9449-53a5cb69376d-plugins-config-volume" pod "69a516d6-a843-40be-9449-53a5cb69376d" (UID: "69a516d6-a843-40be-9449-53a5cb69376d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:23.991098 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.991080    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30a5aa39-82cd-479f-a09b-cab7db69e4ae-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:23.991194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.991105    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/69a516d6-a843-40be-9449-53a5cb69376d-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:23.993365 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.993340    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30a5aa39-82cd-479f-a09b-cab7db69e4ae-kube-api-access-429z6" pod "30a5aa39-82cd-479f-a09b-cab7db69e4ae" (UID: "30a5aa39-82cd-479f-a09b-cab7db69e4ae"). InnerVolumeSpecName "kube-api-access-429z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:23.993365 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:23.993357    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a516d6-a843-40be-9449-53a5cb69376d-kube-api-access-6jbbx" pod "69a516d6-a843-40be-9449-53a5cb69376d" (UID: "69a516d6-a843-40be-9449-53a5cb69376d"). InnerVolumeSpecName "kube-api-access-6jbbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:24.092836 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.092808    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jbbx\" (UniqueName: \"kubernetes.io/projected/69a516d6-a843-40be-9449-53a5cb69376d-kube-api-access-6jbbx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:24.092836 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.092833    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-429z6\" (UniqueName: \"kubernetes.io/projected/30a5aa39-82cd-479f-a09b-cab7db69e4ae-kube-api-access-429z6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:24.143689 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.143591    6000 generic.go:356] "Generic (PLEG): container finished" podID="69a516d6-a843-40be-9449-53a5cb69376d" containerID="e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364" exitCode=0
Nov 03 01:11:24.143689 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.143658    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"
Nov 03 01:11:24.143938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.143689    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl" event={"ID":"69a516d6-a843-40be-9449-53a5cb69376d","Type":"ContainerDied","Data":"e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364"}
Nov 03 01:11:24.143938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.143749    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl" event={"ID":"69a516d6-a843-40be-9449-53a5cb69376d","Type":"ContainerDied","Data":"83c812f3452cb9e4f1b2be4d19c0c7e33e020b8f929ce9fcf2e9fa802828ad30"}
Nov 03 01:11:24.143938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.143777    6000 scope.go:122] "RemoveContainer" containerID="e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364"
Nov 03 01:11:24.145690 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.145662    6000 generic.go:356] "Generic (PLEG): container finished" podID="30a5aa39-82cd-479f-a09b-cab7db69e4ae" containerID="8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f" exitCode=0
Nov 03 01:11:24.145805 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.145715    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"
Nov 03 01:11:24.145805 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.145752    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk" event={"ID":"30a5aa39-82cd-479f-a09b-cab7db69e4ae","Type":"ContainerDied","Data":"8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f"}
Nov 03 01:11:24.145805 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.145796    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk" event={"ID":"30a5aa39-82cd-479f-a09b-cab7db69e4ae","Type":"ContainerDied","Data":"6f4cbbaa4369c9d04b77ce5b1772ff88a2c0403c6a79ba44cd9d67df97a5202f"}
Nov 03 01:11:24.155032 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.154979    6000 scope.go:122] "RemoveContainer" containerID="e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364"
Nov 03 01:11:24.155386 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:24.155359    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364\": container with ID starting with e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364 not found: ID does not exist" containerID="e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364"
Nov 03 01:11:24.155484 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.155397    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364"} err="failed to get container status \"e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364\": rpc error: code = NotFound desc = could not find container \"e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364\": container with ID starting with e903e4efb10832eb917d9ec41cc0dde86bbec972512cbb7c098eb18945580364 not found: ID does not exist"
Nov 03 01:11:24.155484 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.155424    6000 scope.go:122] "RemoveContainer" containerID="8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f"
Nov 03 01:11:24.165288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.165269    6000 scope.go:122] "RemoveContainer" containerID="8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f"
Nov 03 01:11:24.165552 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:24.165533    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f\": container with ID starting with 8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f not found: ID does not exist" containerID="8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f"
Nov 03 01:11:24.165603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.165560    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f"} err="failed to get container status \"8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f\": rpc error: code = NotFound desc = could not find container \"8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f\": container with ID starting with 8581ea6c53064ed30f3755e53e2624911744997f79b88689f86d0f65f77cb70f not found: ID does not exist"
Nov 03 01:11:24.197988 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.196716    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"]
Nov 03 01:11:24.210446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.205314    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-c6e808d7-7bc6ffddf5-s7hnl"]
Nov 03 01:11:24.242241 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.242209    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"]
Nov 03 01:11:24.246001 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.245973    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-c6e808d7-6fd7b8946f-jctwk"]
Nov 03 01:11:24.692887 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.692851    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30a5aa39-82cd-479f-a09b-cab7db69e4ae" path="/var/lib/kubelet/pods/30a5aa39-82cd-479f-a09b-cab7db69e4ae/volumes"
Nov 03 01:11:24.693513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.693485    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a516d6-a843-40be-9449-53a5cb69376d" path="/var/lib/kubelet/pods/69a516d6-a843-40be-9449-53a5cb69376d/volumes"
Nov 03 01:11:24.964322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.964208    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"]
Nov 03 01:11:24.964506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:24.964442    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw" podUID="80e4c79f-9376-4904-9d4e-6fbb7208413a" containerName="maas-api" containerID="cri-o://d803e8c16c7fa90244ab6d52a58bddd0e27295a6566d90c7eb568c131dfeb4ce" gracePeriod=30
Nov 03 01:11:25.162443 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.162409    6000 generic.go:356] "Generic (PLEG): container finished" podID="80e4c79f-9376-4904-9d4e-6fbb7208413a" containerID="d803e8c16c7fa90244ab6d52a58bddd0e27295a6566d90c7eb568c131dfeb4ce" exitCode=0
Nov 03 01:11:25.162628 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.162499    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw" event={"ID":"80e4c79f-9376-4904-9d4e-6fbb7208413a","Type":"ContainerDied","Data":"d803e8c16c7fa90244ab6d52a58bddd0e27295a6566d90c7eb568c131dfeb4ce"}
Nov 03 01:11:25.307699 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.307669    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 01:11:25.452482 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.452448    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80e4c79f-9376-4904-9d4e-6fbb7208413a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e4c79f-9376-4904-9d4e-6fbb7208413a-openshift-service-ca\") pod \"80e4c79f-9376-4904-9d4e-6fbb7208413a\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") "
Nov 03 01:11:25.452685 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.452566    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-maas-api-tls\") pod \"80e4c79f-9376-4904-9d4e-6fbb7208413a\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") "
Nov 03 01:11:25.452685 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.452643    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-metrics-certs\") pod \"80e4c79f-9376-4904-9d4e-6fbb7208413a\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") "
Nov 03 01:11:25.452850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.452775    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80e4c79f-9376-4904-9d4e-6fbb7208413a-kube-api-access-9hlw4\" (UniqueName: \"kubernetes.io/projected/80e4c79f-9376-4904-9d4e-6fbb7208413a-kube-api-access-9hlw4\") pod \"80e4c79f-9376-4904-9d4e-6fbb7208413a\" (UID: \"80e4c79f-9376-4904-9d4e-6fbb7208413a\") "
Nov 03 01:11:25.458601 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.458568    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80e4c79f-9376-4904-9d4e-6fbb7208413a-openshift-service-ca" pod "80e4c79f-9376-4904-9d4e-6fbb7208413a" (UID: "80e4c79f-9376-4904-9d4e-6fbb7208413a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:25.486720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.482021    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-metrics-certs" pod "80e4c79f-9376-4904-9d4e-6fbb7208413a" (UID: "80e4c79f-9376-4904-9d4e-6fbb7208413a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:25.489844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.489775    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-maas-api-tls" pod "80e4c79f-9376-4904-9d4e-6fbb7208413a" (UID: "80e4c79f-9376-4904-9d4e-6fbb7208413a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:25.490481 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.490450    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80e4c79f-9376-4904-9d4e-6fbb7208413a-kube-api-access-9hlw4" pod "80e4c79f-9376-4904-9d4e-6fbb7208413a" (UID: "80e4c79f-9376-4904-9d4e-6fbb7208413a"). InnerVolumeSpecName "kube-api-access-9hlw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:25.554834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.554750    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hlw4\" (UniqueName: \"kubernetes.io/projected/80e4c79f-9376-4904-9d4e-6fbb7208413a-kube-api-access-9hlw4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:25.554834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.554782    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80e4c79f-9376-4904-9d4e-6fbb7208413a-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:25.554834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.554793    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:25.554834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:25.554802    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80e4c79f-9376-4904-9d4e-6fbb7208413a-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:26.172451 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:26.172426    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"
Nov 03 01:11:26.172915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:26.172426    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw" event={"ID":"80e4c79f-9376-4904-9d4e-6fbb7208413a","Type":"ContainerDied","Data":"2445b9b498961c37ef4ab4d24ca04f07cca27735176debaa49aa0e765421697d"}
Nov 03 01:11:26.172915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:26.172561    6000 scope.go:122] "RemoveContainer" containerID="d803e8c16c7fa90244ab6d52a58bddd0e27295a6566d90c7eb568c131dfeb4ce"
Nov 03 01:11:26.220423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:26.220390    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"]
Nov 03 01:11:26.231969 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:26.231944    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-2858fb-6b8777767f-f76qw"]
Nov 03 01:11:26.574999 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:26.574915    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80e4c79f-9376-4904-9d4e-6fbb7208413a" path="/var/lib/kubelet/pods/80e4c79f-9376-4904-9d4e-6fbb7208413a/volumes"
Nov 03 01:11:27.913675 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:27.913635    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:11:31.712342 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:31.712304    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"]
Nov 03 01:11:35.388628 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:35.388589    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"]
Nov 03 01:11:36.000712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.000669    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk" podUID="94c479c0-25ca-49b6-b99d-55a39d17cc52" containerName="main" containerID="cri-o://60377d499bf149c9a8f20caff2df16103a6ec643145663af57d7b2a0d47832f0" gracePeriod=45
Nov 03 01:11:36.260235 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.260207    6000 generic.go:356] "Generic (PLEG): container finished" podID="94c479c0-25ca-49b6-b99d-55a39d17cc52" containerID="60377d499bf149c9a8f20caff2df16103a6ec643145663af57d7b2a0d47832f0" exitCode=0
Nov 03 01:11:36.260390 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.260357    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk" event={"ID":"94c479c0-25ca-49b6-b99d-55a39d17cc52","Type":"ContainerDied","Data":"60377d499bf149c9a8f20caff2df16103a6ec643145663af57d7b2a0d47832f0"}
Nov 03 01:11:36.260472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.260409    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk" event={"ID":"94c479c0-25ca-49b6-b99d-55a39d17cc52","Type":"ContainerDied","Data":"c4e1eeb43b08b10c50b8e1672ba2c23a183a8c29a086ef111f1b7a4a483c92d3"}
Nov 03 01:11:36.260472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.260426    6000 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c4e1eeb43b08b10c50b8e1672ba2c23a183a8c29a086ef111f1b7a4a483c92d3"
Nov 03 01:11:36.282649 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.282620    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:11:36.339619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.339584    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-tmp-dir\") pod \"94c479c0-25ca-49b6-b99d-55a39d17cc52\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") "
Nov 03 01:11:36.339785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.339715    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-dshm\") pod \"94c479c0-25ca-49b6-b99d-55a39d17cc52\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") "
Nov 03 01:11:36.339785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.339752    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94c479c0-25ca-49b6-b99d-55a39d17cc52-tls-certs\" (UniqueName: \"kubernetes.io/secret/94c479c0-25ca-49b6-b99d-55a39d17cc52-tls-certs\") pod \"94c479c0-25ca-49b6-b99d-55a39d17cc52\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") "
Nov 03 01:11:36.339918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.339789    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-model-cache\") pod \"94c479c0-25ca-49b6-b99d-55a39d17cc52\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") "
Nov 03 01:11:36.339918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.339856    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-tmp-dir" pod "94c479c0-25ca-49b6-b99d-55a39d17cc52" (UID: "94c479c0-25ca-49b6-b99d-55a39d17cc52"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:36.340065 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.340011    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94c479c0-25ca-49b6-b99d-55a39d17cc52-kube-api-access-fpglf\" (UniqueName: \"kubernetes.io/projected/94c479c0-25ca-49b6-b99d-55a39d17cc52-kube-api-access-fpglf\") pod \"94c479c0-25ca-49b6-b99d-55a39d17cc52\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") "
Nov 03 01:11:36.340143 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.340128    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-home\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-home\") pod \"94c479c0-25ca-49b6-b99d-55a39d17cc52\" (UID: \"94c479c0-25ca-49b6-b99d-55a39d17cc52\") "
Nov 03 01:11:36.340385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.340128    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-model-cache" pod "94c479c0-25ca-49b6-b99d-55a39d17cc52" (UID: "94c479c0-25ca-49b6-b99d-55a39d17cc52"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:36.340599 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.340346    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-home" pod "94c479c0-25ca-49b6-b99d-55a39d17cc52" (UID: "94c479c0-25ca-49b6-b99d-55a39d17cc52"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:36.340757 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.340735    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:36.340917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.340898    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:36.343402 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.343373    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94c479c0-25ca-49b6-b99d-55a39d17cc52-tls-certs" pod "94c479c0-25ca-49b6-b99d-55a39d17cc52" (UID: "94c479c0-25ca-49b6-b99d-55a39d17cc52"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:36.343402 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.343389    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-dshm" pod "94c479c0-25ca-49b6-b99d-55a39d17cc52" (UID: "94c479c0-25ca-49b6-b99d-55a39d17cc52"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:36.343588 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.343425    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94c479c0-25ca-49b6-b99d-55a39d17cc52-kube-api-access-fpglf" pod "94c479c0-25ca-49b6-b99d-55a39d17cc52" (UID: "94c479c0-25ca-49b6-b99d-55a39d17cc52"). InnerVolumeSpecName "kube-api-access-fpglf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:36.442014 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.441982    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:36.442443 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.442013    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/94c479c0-25ca-49b6-b99d-55a39d17cc52-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:36.442443 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.442036    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fpglf\" (UniqueName: \"kubernetes.io/projected/94c479c0-25ca-49b6-b99d-55a39d17cc52-kube-api-access-fpglf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:36.442443 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.442044    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/94c479c0-25ca-49b6-b99d-55a39d17cc52-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:36.712676 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:36.712626    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" podUID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" containerName="payload-pre-processing" containerID="cri-o://13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f" gracePeriod=25
Nov 03 01:11:37.266959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:37.266929    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"
Nov 03 01:11:37.318292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:37.318234    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"]
Nov 03 01:11:37.329761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:37.329728    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/rate-test-model-9908e8-kserve-5fbf6fb6cc-v4vnk"]
Nov 03 01:11:38.567946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:38.567913    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94c479c0-25ca-49b6-b99d-55a39d17cc52" path="/var/lib/kubelet/pods/94c479c0-25ca-49b6-b99d-55a39d17cc52/volumes"
Nov 03 01:11:44.794407 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:44.794366    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" podUID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:49.896992 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:49.896939    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:11:50.411182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.411139    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main" containerID="cri-o://af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1" gracePeriod=45
Nov 03 01:11:50.695878 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.695852    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 01:11:50.780993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.780961    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/141ac223-111f-450f-b637-126126d23bd5-kube-api-access-cxlbw\" (UniqueName: \"kubernetes.io/projected/141ac223-111f-450f-b637-126126d23bd5-kube-api-access-cxlbw\") pod \"141ac223-111f-450f-b637-126126d23bd5\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") "
Nov 03 01:11:50.781175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.781044    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-dshm\") pod \"141ac223-111f-450f-b637-126126d23bd5\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") "
Nov 03 01:11:50.781175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.781119    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-home\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-home\") pod \"141ac223-111f-450f-b637-126126d23bd5\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") "
Nov 03 01:11:50.781288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.781193    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-tmp-dir\") pod \"141ac223-111f-450f-b637-126126d23bd5\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") "
Nov 03 01:11:50.781353 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.781288    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-model-cache\") pod \"141ac223-111f-450f-b637-126126d23bd5\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") "
Nov 03 01:11:50.781480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.781454    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/141ac223-111f-450f-b637-126126d23bd5-tls-certs\" (UniqueName: \"kubernetes.io/secret/141ac223-111f-450f-b637-126126d23bd5-tls-certs\") pod \"141ac223-111f-450f-b637-126126d23bd5\" (UID: \"141ac223-111f-450f-b637-126126d23bd5\") "
Nov 03 01:11:50.781882 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.781822    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-home" pod "141ac223-111f-450f-b637-126126d23bd5" (UID: "141ac223-111f-450f-b637-126126d23bd5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.781882 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.781852    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-model-cache" pod "141ac223-111f-450f-b637-126126d23bd5" (UID: "141ac223-111f-450f-b637-126126d23bd5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.782119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.782094    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.782239 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.782122    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.782239 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.782186    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-tmp-dir" pod "141ac223-111f-450f-b637-126126d23bd5" (UID: "141ac223-111f-450f-b637-126126d23bd5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.784718 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.784676    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/141ac223-111f-450f-b637-126126d23bd5-tls-certs" pod "141ac223-111f-450f-b637-126126d23bd5" (UID: "141ac223-111f-450f-b637-126126d23bd5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:50.784893 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.784869    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-dshm" pod "141ac223-111f-450f-b637-126126d23bd5" (UID: "141ac223-111f-450f-b637-126126d23bd5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:11:50.784987 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.784904    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/141ac223-111f-450f-b637-126126d23bd5-kube-api-access-cxlbw" pod "141ac223-111f-450f-b637-126126d23bd5" (UID: "141ac223-111f-450f-b637-126126d23bd5"). InnerVolumeSpecName "kube-api-access-cxlbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:50.883963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.883929    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/141ac223-111f-450f-b637-126126d23bd5-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.883963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.883962    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxlbw\" (UniqueName: \"kubernetes.io/projected/141ac223-111f-450f-b637-126126d23bd5-kube-api-access-cxlbw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.883963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.883970    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:50.884314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:50.883980    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/141ac223-111f-450f-b637-126126d23bd5-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:51.370849 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.370807    6000 generic.go:356] "Generic (PLEG): container finished" podID="141ac223-111f-450f-b637-126126d23bd5" containerID="af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1" exitCode=0
Nov 03 01:11:51.371382 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.370867    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"
Nov 03 01:11:51.371382 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.370898    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" event={"ID":"141ac223-111f-450f-b637-126126d23bd5","Type":"ContainerDied","Data":"af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1"}
Nov 03 01:11:51.371382 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.370945    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn" event={"ID":"141ac223-111f-450f-b637-126126d23bd5","Type":"ContainerDied","Data":"f57792832a7db01b19131dae3fd968ad3cf5a824856279a204b7b38af2b2f09d"}
Nov 03 01:11:51.371382 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.370962    6000 scope.go:122] "RemoveContainer" containerID="af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1"
Nov 03 01:11:51.382058 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.382037    6000 scope.go:122] "RemoveContainer" containerID="af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1"
Nov 03 01:11:51.382349 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:51.382325    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1\": container with ID starting with af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1 not found: ID does not exist" containerID="af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1"
Nov 03 01:11:51.382445 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.382359    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1"} err="failed to get container status \"af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1\": rpc error: code = NotFound desc = could not find container \"af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1\": container with ID starting with af1f150913be817f4a939cc1b606afc0316d0b0391ca19ba35f29e0c830963d1 not found: ID does not exist"
Nov 03 01:11:51.418718 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.418690    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"]
Nov 03 01:11:51.429784 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:51.429761    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8e90c5/distinct-w3-8e90c5-kserve-69d67cbf48-hlzsn"]
Nov 03 01:11:52.569113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:52.569079    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="141ac223-111f-450f-b637-126126d23bd5" path="/var/lib/kubelet/pods/141ac223-111f-450f-b637-126126d23bd5/volumes"
Nov 03 01:11:54.794239 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:54.794203    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" podUID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:11:55.835947 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.835835    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"]
Nov 03 01:11:55.836775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.836749    6000 cpu_manager.go:414] "containerMap: removing container" podUID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" containerName="payload-pre-processing"
Nov 03 01:11:55.836957 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.836931    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" containerName="payload-pre-processing"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.836962    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d17d7d40-68cf-482f-86fa-34f5af94813c" containerName="payload-processing"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.836969    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17d7d40-68cf-482f-86fa-34f5af94813c" containerName="payload-processing"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.836983    6000 cpu_manager.go:414] "containerMap: removing container" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.836988    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.836998    6000 cpu_manager.go:414] "containerMap: removing container" podUID="30a5aa39-82cd-479f-a09b-cab7db69e4ae" containerName="payload-processing"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837003    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30a5aa39-82cd-479f-a09b-cab7db69e4ae" containerName="payload-processing"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837018    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2f71ee87-6e9f-4e44-b400-577011399180" containerName="payload-pre-processing"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837025    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f71ee87-6e9f-4e44-b400-577011399180" containerName="payload-pre-processing"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837041    6000 cpu_manager.go:414] "containerMap: removing container" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837046    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837056    6000 cpu_manager.go:414] "containerMap: removing container" podUID="80e4c79f-9376-4904-9d4e-6fbb7208413a" containerName="maas-api"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837061    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80e4c79f-9376-4904-9d4e-6fbb7208413a" containerName="maas-api"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837068    6000 cpu_manager.go:414] "containerMap: removing container" podUID="94c479c0-25ca-49b6-b99d-55a39d17cc52" containerName="main"
Nov 03 01:11:55.837068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837073    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94c479c0-25ca-49b6-b99d-55a39d17cc52" containerName="main"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837086    6000 cpu_manager.go:414] "containerMap: removing container" podUID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837091    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837102    6000 cpu_manager.go:414] "containerMap: removing container" podUID="69a516d6-a843-40be-9449-53a5cb69376d" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837111    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a516d6-a843-40be-9449-53a5cb69376d" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837118    6000 cpu_manager.go:414] "containerMap: removing container" podUID="40b24645-1feb-4ed5-b735-cb817f512708" containerName="maas-api"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837122    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40b24645-1feb-4ed5-b735-cb817f512708" containerName="maas-api"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837135    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e8c90679-b97d-4691-9fa4-352d916678dd" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837140    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8c90679-b97d-4691-9fa4-352d916678dd" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837148    6000 cpu_manager.go:414] "containerMap: removing container" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837152    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837158    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837163    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837169    6000 cpu_manager.go:414] "containerMap: removing container" podUID="4729ad2b-ba3f-47f0-9cae-7c843a10781f" containerName="maas-api"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837173    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4729ad2b-ba3f-47f0-9cae-7c843a10781f" containerName="maas-api"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837195    6000 cpu_manager.go:414] "containerMap: removing container" podUID="78254f06-bf41-44d1-8c79-b236c12f380e" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837199    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78254f06-bf41-44d1-8c79-b236c12f380e" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837212    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837216    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837360    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8c90679-b97d-4691-9fa4-352d916678dd" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837372    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837381    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e57d6b5-7b1b-4f25-b7ce-edbe2f18a372" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837386    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="4729ad2b-ba3f-47f0-9cae-7c843a10781f" containerName="maas-api"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837394    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="78254f06-bf41-44d1-8c79-b236c12f380e" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837402    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d2f8092-9d61-4b06-a9d4-a304c85d9f8b" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837410    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7811f9d-d30d-4fee-850f-10c22d42e2cc" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837415    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17d7d40-68cf-482f-86fa-34f5af94813c" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837422    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="141ac223-111f-450f-b637-126126d23bd5" containerName="main"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837428    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="30a5aa39-82cd-479f-a09b-cab7db69e4ae" containerName="payload-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837436    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f71ee87-6e9f-4e44-b400-577011399180" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837447    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="80e4c79f-9376-4904-9d4e-6fbb7208413a" containerName="maas-api"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837455    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="94c479c0-25ca-49b6-b99d-55a39d17cc52" containerName="main"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837461    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="aff0a103-896a-43e3-a6b9-c02f8ffa23ff" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837465    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a516d6-a843-40be-9449-53a5cb69376d" containerName="payload-pre-processing"
Nov 03 01:11:55.837598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.837471    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="40b24645-1feb-4ed5-b735-cb817f512708" containerName="maas-api"
Nov 03 01:11:55.843585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.843557    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:55.862768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.862736    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"]
Nov 03 01:11:55.871633 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.871601    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"default-dockercfg-8kwzh\""
Nov 03 01:11:55.873317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.873292    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"openshift-service-ca.crt\""
Nov 03 01:11:55.873903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.873401    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"kube-root-ca.crt\""
Nov 03 01:11:55.873903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.873460    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-9908e8\"/\"sub-test-model-9908e8-kserve-self-signed-certs\""
Nov 03 01:11:55.955238 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.955202    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tls-certs\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:55.955444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.955283    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tmp-dir\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:55.955444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.955360    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-model-cache\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:55.955444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.955423    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-dshm\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:55.955576 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.955562    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r77zs\" (UniqueName: \"kubernetes.io/projected/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-kube-api-access-r77zs\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:55.955688 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.955672    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-home\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:55.975885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.975856    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"]
Nov 03 01:11:55.976161 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:55.976119    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj" podUID="bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" containerName="maas-api" containerID="cri-o://1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459" gracePeriod=30
Nov 03 01:11:56.057475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.057441    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r77zs\" (UniqueName: \"kubernetes.io/projected/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-kube-api-access-r77zs\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.057650 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.057541    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-home\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.057650 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.057610    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tls-certs\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.057650 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.057636    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tmp-dir\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.057844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.057662    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-model-cache\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.057844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.057694    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-dshm\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.058054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.058032    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-home\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.058130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.058066    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tmp-dir\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.058130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.058101    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-model-cache\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.060081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.060059    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-dshm\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.060428 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.060409    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tls-certs\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.076765 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.076740    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r77zs\" (UniqueName: \"kubernetes.io/projected/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-kube-api-access-r77zs\") pod \"sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") " pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.178917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.178857    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:56.318304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.316917    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 01:11:56.387290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.386851    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-kube-api-access-rgphp\" (UniqueName: \"kubernetes.io/projected/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-kube-api-access-rgphp\") pod \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") "
Nov 03 01:11:56.387290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.387017    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-openshift-service-ca\") pod \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") "
Nov 03 01:11:56.387290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.387086    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-metrics-certs\") pod \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") "
Nov 03 01:11:56.387290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.387211    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-maas-api-tls\") pod \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\" (UID: \"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36\") "
Nov 03 01:11:56.394752 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.394704    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-maas-api-tls" pod "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" (UID: "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:56.395225 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.395151    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-openshift-service-ca" pod "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" (UID: "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:56.395933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.395901    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-kube-api-access-rgphp" pod "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" (UID: "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36"). InnerVolumeSpecName "kube-api-access-rgphp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:56.398287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.398199    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-metrics-certs" pod "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" (UID: "bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:11:56.443279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.441982    6000 generic.go:356] "Generic (PLEG): container finished" podID="bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" containerID="1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459" exitCode=0
Nov 03 01:11:56.443279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.442126    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj" event={"ID":"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36","Type":"ContainerDied","Data":"1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459"}
Nov 03 01:11:56.443279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.442181    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj" event={"ID":"bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36","Type":"ContainerDied","Data":"1417e98c7b21ebf90561765f494d3f53c7b69e56d9a5327aa6c878354bdac9d6"}
Nov 03 01:11:56.443279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.442215    6000 scope.go:122] "RemoveContainer" containerID="1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459"
Nov 03 01:11:56.443279 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.442638    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"
Nov 03 01:11:56.491075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.490965    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:56.491075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.491003    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:56.491075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.491017    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgphp\" (UniqueName: \"kubernetes.io/projected/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-kube-api-access-rgphp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:56.491075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.491030    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:56.492318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.492169    6000 scope.go:122] "RemoveContainer" containerID="1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459"
Nov 03 01:11:56.492860 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:56.492766    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459\": container with ID starting with 1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459 not found: ID does not exist" containerID="1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459"
Nov 03 01:11:56.492860 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.492806    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459"} err="failed to get container status \"1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459\": rpc error: code = NotFound desc = could not find container \"1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459\": container with ID starting with 1944f7c2c62bb6fa3b1531a130e064d7dfd8ed2ef03f7adab7ac9c873730b459 not found: ID does not exist"
Nov 03 01:11:56.502918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.502882    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"]
Nov 03 01:11:56.506948 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.506919    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-e62935-6857d45fc6-kvbtj"]
Nov 03 01:11:56.568569 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.568540    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" path="/var/lib/kubelet/pods/bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36/volumes"
Nov 03 01:11:56.577383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:56.577360    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"]
Nov 03 01:11:56.578311 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:11:56.578287    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf1ca0eaa_8944_45a6_bc7f_5da1d50b62a8.slice/crio-de287d236c0933129722c5a6e24f8b99ca160e38ec708fb60fc9e7c06df5efb4 WatchSource:0}: Error finding container de287d236c0933129722c5a6e24f8b99ca160e38ec708fb60fc9e7c06df5efb4: Status 404 returned error can't find the container with id de287d236c0933129722c5a6e24f8b99ca160e38ec708fb60fc9e7c06df5efb4
Nov 03 01:11:57.034801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.034770    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 01:11:57.112497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.112456    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-kube-api-access-vhskp\" (UniqueName: \"kubernetes.io/projected/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-kube-api-access-vhskp\") pod \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") "
Nov 03 01:11:57.112707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.112532    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-plugins-config-volume\") pod \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\" (UID: \"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d\") "
Nov 03 01:11:57.113025 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.112988    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-plugins-config-volume" pod "c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" (UID: "c1c0c4b2-8ae9-40cf-8efe-3d947241b77d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:11:57.114864 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.114835    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-kube-api-access-vhskp" pod "c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" (UID: "c1c0c4b2-8ae9-40cf-8efe-3d947241b77d"). InnerVolumeSpecName "kube-api-access-vhskp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:11:57.214010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.213914    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhskp\" (UniqueName: \"kubernetes.io/projected/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-kube-api-access-vhskp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:57.214010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.213955    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:11:57.450349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.450312    6000 generic.go:356] "Generic (PLEG): container finished" podID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" containerID="13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f" exitCode=0
Nov 03 01:11:57.450532 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.450370    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"
Nov 03 01:11:57.450532 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.450393    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" event={"ID":"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d","Type":"ContainerDied","Data":"13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f"}
Nov 03 01:11:57.450532 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.450449    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh" event={"ID":"c1c0c4b2-8ae9-40cf-8efe-3d947241b77d","Type":"ContainerDied","Data":"c80f58ceb9a40447cdd9667bb0d02420f95314daf5b9c4c810a7b1e02e11accc"}
Nov 03 01:11:57.450532 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.450472    6000 scope.go:122] "RemoveContainer" containerID="13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f"
Nov 03 01:11:57.452644 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.452620    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq" event={"ID":"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8","Type":"ContainerStarted","Data":"9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681"}
Nov 03 01:11:57.453322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.452659    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq" event={"ID":"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8","Type":"ContainerStarted","Data":"de287d236c0933129722c5a6e24f8b99ca160e38ec708fb60fc9e7c06df5efb4"}
Nov 03 01:11:57.462210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.462188    6000 scope.go:122] "RemoveContainer" containerID="13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f"
Nov 03 01:11:57.462513 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:11:57.462494    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f\": container with ID starting with 13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f not found: ID does not exist" containerID="13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f"
Nov 03 01:11:57.462593 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.462527    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f"} err="failed to get container status \"13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f\": rpc error: code = NotFound desc = could not find container \"13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f\": container with ID starting with 13e97ac52a91d0131be2eefcdd020ba15583a7b4e4e23e145f1bd31b1f4b1e5f not found: ID does not exist"
Nov 03 01:11:57.531094 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.530968    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq" podStartSLOduration=2.5309514010000003 podStartE2EDuration="2.530951401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:11:57.530418704 +0000 UTC m=+3955.085770731" watchObservedRunningTime="2026-11-03 01:11:57.530951401 +0000 UTC m=+3955.086303425"
Nov 03 01:11:57.589962 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.589929    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"]
Nov 03 01:11:57.594845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:57.594815    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e62935-76c4db8dc-5p6lh"]
Nov 03 01:11:58.179788 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.179757    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:11:58.568244 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.568166    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" path="/var/lib/kubelet/pods/c1c0c4b2-8ae9-40cf-8efe-3d947241b77d/volumes"
Nov 03 01:11:58.800932 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.800896    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"]
Nov 03 01:11:58.801694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.801675    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" containerName="payload-pre-processing"
Nov 03 01:11:58.801768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.801695    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" containerName="payload-pre-processing"
Nov 03 01:11:58.801768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.801720    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" containerName="maas-api"
Nov 03 01:11:58.801768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.801725    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" containerName="maas-api"
Nov 03 01:11:58.801890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.801832    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1c0c4b2-8ae9-40cf-8efe-3d947241b77d" containerName="payload-pre-processing"
Nov 03 01:11:58.801890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.801845    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bcd0ac0b-733e-4b26-ae0d-1ccacdaeeb36" containerName="maas-api"
Nov 03 01:11:58.801890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.801854    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="587e6577-0386-4e2c-85e2-8e91b40c5fbc" containerName="istio-proxy"
Nov 03 01:11:58.807171 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.807141    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:58.826416 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.826352    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"]
Nov 03 01:11:58.832458 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.832432    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-099583\"/\"kube-root-ca.crt\""
Nov 03 01:11:58.832458 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.832434    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-099583\"/\"openshift-service-ca.crt\""
Nov 03 01:11:58.832654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.832519    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-099583\"/\"default-dockercfg-fplj7\""
Nov 03 01:11:58.834029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.834008    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-099583\"/\"sub-test-model-099583-kserve-self-signed-certs\""
Nov 03 01:11:58.937742 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.937707    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-home\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:58.937936 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.937900    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tls-certs\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:58.938105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.937947    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-model-cache\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:58.938105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.937975    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tmp-dir\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:58.938105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.938046    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfnpx\" (UniqueName: \"kubernetes.io/projected/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-kube-api-access-sfnpx\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:58.938351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.938233    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-dshm\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:58.976375 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:58.976346    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:11:59.050281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.049170    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-home\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.050281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.049395    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tls-certs\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.050281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.049455    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-model-cache\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.050281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.049495    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tmp-dir\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.050281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.049530    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfnpx\" (UniqueName: \"kubernetes.io/projected/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-kube-api-access-sfnpx\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.050281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.049652    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-dshm\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.051181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.050518    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-model-cache\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.051181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.050763    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-home\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.066543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.051461    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tmp-dir\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.071686 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.067655    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-dshm\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.077056 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.076964    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfnpx\" (UniqueName: \"kubernetes.io/projected/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-kube-api-access-sfnpx\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.082505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.079239    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tls-certs\") pod \"sub-test-model-099583-kserve-596b4695f7-jjtmt\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") " pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.126180 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.126128    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:11:59.429316 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:11:59.429287    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54d2cebd_68c0_4b15_ac8e_5088762e5cdf.slice/crio-0d75f92da6a4c400028c52e88cc3e63281026dd549bcf7e3637b506656df0558 WatchSource:0}: Error finding container 0d75f92da6a4c400028c52e88cc3e63281026dd549bcf7e3637b506656df0558: Status 404 returned error can't find the container with id 0d75f92da6a4c400028c52e88cc3e63281026dd549bcf7e3637b506656df0558
Nov 03 01:11:59.429754 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.429720    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"]
Nov 03 01:11:59.472637 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:11:59.472608    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt" event={"ID":"54d2cebd-68c0-4b15-ac8e-5088762e5cdf","Type":"ContainerStarted","Data":"0d75f92da6a4c400028c52e88cc3e63281026dd549bcf7e3637b506656df0558"}
Nov 03 01:12:00.064645 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.064602    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"]
Nov 03 01:12:00.064933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.064901    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" podUID="651e7bfb-6645-492e-99aa-1721bd7a6322" containerName="maas-api" containerID="cri-o://5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b" gracePeriod=30
Nov 03 01:12:00.391697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.391669    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 01:12:00.474718 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.474688    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-metrics-certs\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-metrics-certs\") pod \"651e7bfb-6645-492e-99aa-1721bd7a6322\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") "
Nov 03 01:12:00.475220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.474814    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651e7bfb-6645-492e-99aa-1721bd7a6322-kube-api-access-kgz5n\" (UniqueName: \"kubernetes.io/projected/651e7bfb-6645-492e-99aa-1721bd7a6322-kube-api-access-kgz5n\") pod \"651e7bfb-6645-492e-99aa-1721bd7a6322\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") "
Nov 03 01:12:00.475220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.474852    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls\") pod \"651e7bfb-6645-492e-99aa-1721bd7a6322\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") "
Nov 03 01:12:00.475220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.475115    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651e7bfb-6645-492e-99aa-1721bd7a6322-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651e7bfb-6645-492e-99aa-1721bd7a6322-openshift-service-ca\") pod \"651e7bfb-6645-492e-99aa-1721bd7a6322\" (UID: \"651e7bfb-6645-492e-99aa-1721bd7a6322\") "
Nov 03 01:12:00.480727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.480690    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651e7bfb-6645-492e-99aa-1721bd7a6322-openshift-service-ca" pod "651e7bfb-6645-492e-99aa-1721bd7a6322" (UID: "651e7bfb-6645-492e-99aa-1721bd7a6322"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:00.480883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.480804    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-metrics-certs" pod "651e7bfb-6645-492e-99aa-1721bd7a6322" (UID: "651e7bfb-6645-492e-99aa-1721bd7a6322"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:00.481674 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.481644    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls" pod "651e7bfb-6645-492e-99aa-1721bd7a6322" (UID: "651e7bfb-6645-492e-99aa-1721bd7a6322"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:00.483873 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.483836    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651e7bfb-6645-492e-99aa-1721bd7a6322-kube-api-access-kgz5n" pod "651e7bfb-6645-492e-99aa-1721bd7a6322" (UID: "651e7bfb-6645-492e-99aa-1721bd7a6322"). InnerVolumeSpecName "kube-api-access-kgz5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:00.485899 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.485864    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt" event={"ID":"54d2cebd-68c0-4b15-ac8e-5088762e5cdf","Type":"ContainerStarted","Data":"04a636572fa2f9daa30e0d8b02028198206cebd0a967660499c93f482ce43006"}
Nov 03 01:12:00.488050 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.488023    6000 generic.go:356] "Generic (PLEG): container finished" podID="651e7bfb-6645-492e-99aa-1721bd7a6322" containerID="5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b" exitCode=0
Nov 03 01:12:00.488154 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.488069    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"
Nov 03 01:12:00.488154 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.488107    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" event={"ID":"651e7bfb-6645-492e-99aa-1721bd7a6322","Type":"ContainerDied","Data":"5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b"}
Nov 03 01:12:00.488328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.488192    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" event={"ID":"651e7bfb-6645-492e-99aa-1721bd7a6322","Type":"ContainerDied","Data":"a3778c65322023496d9b45f94392f6cbde4b5e733c928c6b5ee152e1ae3c086b"}
Nov 03 01:12:00.488328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.488228    6000 scope.go:122] "RemoveContainer" containerID="5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b"
Nov 03 01:12:00.509364 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.509339    6000 scope.go:122] "RemoveContainer" containerID="5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b"
Nov 03 01:12:00.509979 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:00.509956    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b\": container with ID starting with 5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b not found: ID does not exist" containerID="5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b"
Nov 03 01:12:00.510079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.509988    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b"} err="failed to get container status \"5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b\": rpc error: code = NotFound desc = could not find container \"5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b\": container with ID starting with 5e5fad9e4cf493c993db9ae460eed6ff50f48dd1ba0dd2a617af584c2c6e855b not found: ID does not exist"
Nov 03 01:12:00.532664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.532600    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt" podStartSLOduration=2.532582596 podStartE2EDuration="2.532582596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:12:00.529866008 +0000 UTC m=+3958.085218046" watchObservedRunningTime="2026-11-03 01:12:00.532582596 +0000 UTC m=+3958.087934621"
Nov 03 01:12:00.569405 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.569378    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"]
Nov 03 01:12:00.569579 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.569416    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx"]
Nov 03 01:12:00.577603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.577574    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:00.577603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.577603    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgz5n\" (UniqueName: \"kubernetes.io/projected/651e7bfb-6645-492e-99aa-1721bd7a6322-kube-api-access-kgz5n\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:00.577794 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.577625    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/651e7bfb-6645-492e-99aa-1721bd7a6322-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:00.577794 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:00.577640    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/651e7bfb-6645-492e-99aa-1721bd7a6322-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:01.127082 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:01.127042    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:12:02.570155 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:02.570126    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651e7bfb-6645-492e-99aa-1721bd7a6322" path="/var/lib/kubelet/pods/651e7bfb-6645-492e-99aa-1721bd7a6322/volumes"
Nov 03 01:12:03.384340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:03.384288    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-8e90c5-7b889c74-7c9bx" podUID="651e7bfb-6645-492e-99aa-1721bd7a6322" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.53:8443/health\": context deadline exceeded"
Nov 03 01:12:06.179918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:06.179880    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:12:06.193280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:06.193221    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:12:06.554461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:06.554375    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:12:09.127061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:09.127025    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:12:09.140022 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:09.139996    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:12:09.576937 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:09.576845    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:12:11.192740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:11.192705    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"]
Nov 03 01:12:11.226686 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:11.226651    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"]
Nov 03 01:12:14.330053 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.329960    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"]
Nov 03 01:12:14.330891 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.330862    6000 cpu_manager.go:414] "containerMap: removing container" podUID="651e7bfb-6645-492e-99aa-1721bd7a6322" containerName="maas-api"
Nov 03 01:12:14.330891 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.330892    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651e7bfb-6645-492e-99aa-1721bd7a6322" containerName="maas-api"
Nov 03 01:12:14.331073 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.331045    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="651e7bfb-6645-492e-99aa-1721bd7a6322" containerName="maas-api"
Nov 03 01:12:14.334773 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.334753    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.362426 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.362382    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"openshift-service-ca.crt\""
Nov 03 01:12:14.362626 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.362453    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"default-dockercfg-d8kkj\""
Nov 03 01:12:14.362626 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.362514    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"sub-test-model-20109d-kserve-self-signed-certs\""
Nov 03 01:12:14.362626 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.362569    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-20109d\"/\"kube-root-ca.crt\""
Nov 03 01:12:14.366055 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.366028    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"]
Nov 03 01:12:14.432039 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.431996    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tmp-dir\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.432274 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.432130    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-model-cache\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.432274 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.432202    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-dshm\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.432274 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.432231    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tls-certs\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.432462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.432356    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dckns\" (UniqueName: \"kubernetes.io/projected/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-kube-api-access-dckns\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.432523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.432485    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-home\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.533688 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.533646    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tmp-dir\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.533908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.533746    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-model-cache\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.533908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.533773    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-dshm\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.533908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.533796    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tls-certs\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.533908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.533870    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dckns\" (UniqueName: \"kubernetes.io/projected/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-kube-api-access-dckns\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.534130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.533986    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-home\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.534130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.534066    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tmp-dir\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.534343 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.534163    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-model-cache\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.534400 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.534370    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-home\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.536359 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.536330    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-dshm\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.536581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.536565    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tls-certs\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.555704 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.555676    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dckns\" (UniqueName: \"kubernetes.io/projected/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-kube-api-access-dckns\") pod \"sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") " pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.662364 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.658990    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:14.960808 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.959812    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"]
Nov 03 01:12:14.960808 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.960071    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy" containerID="cri-o://816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400" gracePeriod=30
Nov 03 01:12:14.997742 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:14.997711    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"]
Nov 03 01:12:15.678633 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:15.678597    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m" event={"ID":"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7","Type":"ContainerStarted","Data":"99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26"}
Nov 03 01:12:15.679162 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:15.678646    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m" event={"ID":"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7","Type":"ContainerStarted","Data":"35079960df13188284e5e9966f092ea9ae56860fa3be17d6e82824ce3dc43fa1"}
Nov 03 01:12:15.716184 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:15.716122    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m" podStartSLOduration=1.716105227 podStartE2EDuration="1.716105227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:12:15.715584138 +0000 UTC m=+3973.270936173" watchObservedRunningTime="2026-11-03 01:12:15.716105227 +0000 UTC m=+3973.271457250"
Nov 03 01:12:16.194834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:16.194749    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" podUID="ee39d2b9-9e10-4947-b669-300e04a03415" containerName="payload-processing" containerID="cri-o://50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca" gracePeriod=25
Nov 03 01:12:16.227469 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:16.227398    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" podUID="21af1f82-17cb-412b-856e-9f2592be3b4e" containerName="payload-pre-processing" containerID="cri-o://9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20" gracePeriod=25
Nov 03 01:12:16.659866 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:16.659834    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:19.896044 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:19.896010    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:12:20.251181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.251154    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 01:12:20.319971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.319663    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" podUID="21af1f82-17cb-412b-856e-9f2592be3b4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:12:20.319971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.319867    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" podUID="ee39d2b9-9e10-4947-b669-300e04a03415" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:12:20.348576 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.348539    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d17540b0-b06d-48f4-bc53-628863213789-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17540b0-b06d-48f4-bc53-628863213789-istiod-ca-cert\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.348775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.348615    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-istio-token\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-istio-token\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.348775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.348648    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-socket\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.348922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.348776    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-certs\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.348922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.348829    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-envoy\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.348922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.348898    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-kube-api-access-xsqxs\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-kube-api-access-xsqxs\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.349092 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.348960    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d17540b0-b06d-48f4-bc53-628863213789-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17540b0-b06d-48f4-bc53-628863213789-istio-podinfo\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.349092 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349012    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-socket" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:20.349092 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349050    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d17540b0-b06d-48f4-bc53-628863213789-istiod-ca-cert" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:20.349092 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349057    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-credential-socket\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.349336 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349135    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-data\") pod \"d17540b0-b06d-48f4-bc53-628863213789\" (UID: \"d17540b0-b06d-48f4-bc53-628863213789\") "
Nov 03 01:12:20.349336 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349310    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-credential-socket" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:20.349582 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349559    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-certs" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:20.349681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349639    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-data" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:20.349939 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349914    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.350003 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349944    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.350003 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349959    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.350003 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349973    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.350003 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.349988    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d17540b0-b06d-48f4-bc53-628863213789-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.351762 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.351737    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-envoy" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:20.351957 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.351927    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-kube-api-access-xsqxs" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "kube-api-access-xsqxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:20.352086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.352028    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-istio-token" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:20.352163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.352115    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d17540b0-b06d-48f4-bc53-628863213789-istio-podinfo" pod "d17540b0-b06d-48f4-bc53-628863213789" (UID: "d17540b0-b06d-48f4-bc53-628863213789"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:12:20.451905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.451875    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.451905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.451902    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d17540b0-b06d-48f4-bc53-628863213789-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.451905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.451911    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsqxs\" (UniqueName: \"kubernetes.io/projected/d17540b0-b06d-48f4-bc53-628863213789-kube-api-access-xsqxs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.452144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.451924    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d17540b0-b06d-48f4-bc53-628863213789-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:20.717143 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.717106    6000 generic.go:356] "Generic (PLEG): container finished" podID="d17540b0-b06d-48f4-bc53-628863213789" containerID="816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400" exitCode=0
Nov 03 01:12:20.717356 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.717174    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"
Nov 03 01:12:20.717356 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.717195    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" event={"ID":"d17540b0-b06d-48f4-bc53-628863213789","Type":"ContainerDied","Data":"816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400"}
Nov 03 01:12:20.717356 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.717283    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn" event={"ID":"d17540b0-b06d-48f4-bc53-628863213789","Type":"ContainerDied","Data":"ecc800562d7ae82bd05e56277e9dc04fc46f860f79f20b9c7990d9b85c065b64"}
Nov 03 01:12:20.717356 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.717315    6000 scope.go:122] "RemoveContainer" containerID="816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400"
Nov 03 01:12:20.727364 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.727345    6000 scope.go:122] "RemoveContainer" containerID="816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400"
Nov 03 01:12:20.727653 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:20.727628    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400\": container with ID starting with 816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400 not found: ID does not exist" containerID="816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400"
Nov 03 01:12:20.727720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.727653    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400"} err="failed to get container status \"816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400\": rpc error: code = NotFound desc = could not find container \"816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400\": container with ID starting with 816b996089fa9e0e8ae8edf8e4c80baa01626bf7d10a9661ed2c5dc7f17ec400 not found: ID does not exist"
Nov 03 01:12:20.778693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.778660    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"]
Nov 03 01:12:20.790799 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:20.790774    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8e90c5-openshift-default-85b7d7485f-bb5hn"]
Nov 03 01:12:22.569819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:22.569790    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d17540b0-b06d-48f4-bc53-628863213789" path="/var/lib/kubelet/pods/d17540b0-b06d-48f4-bc53-628863213789/volumes"
Nov 03 01:12:24.659685 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:24.659651    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:24.673054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:24.673029    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:24.755391 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:24.755357    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:12:30.320229 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:30.320190    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" podUID="21af1f82-17cb-412b-856e-9f2592be3b4e" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:12:30.320727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:30.320702    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" podUID="ee39d2b9-9e10-4947-b669-300e04a03415" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:12:33.633578 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.633544    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"]
Nov 03 01:12:33.634788 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.634762    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy"
Nov 03 01:12:33.634788 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.634788    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy"
Nov 03 01:12:33.635006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.634982    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d17540b0-b06d-48f4-bc53-628863213789" containerName="istio-proxy"
Nov 03 01:12:33.640537 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.640511    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.662026 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.661995    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"]
Nov 03 01:12:33.669663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.669633    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c109b\"/\"sub-test-model-5c109b-kserve-self-signed-certs\""
Nov 03 01:12:33.670348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.670325    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c109b\"/\"default-dockercfg-9k277\""
Nov 03 01:12:33.670491 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.670397    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c109b\"/\"openshift-service-ca.crt\""
Nov 03 01:12:33.670491 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.670475    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-5c109b\"/\"kube-root-ca.crt\""
Nov 03 01:12:33.737524 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.737491    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-tmp-dir\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.737709 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.737556    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-dshm\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.737709 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.737664    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432f27-b3ec-443d-a298-9c1aea8b27f4-tls-certs\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.737824 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.737776    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-model-cache\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.737824 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.737800    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-home\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.737974 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.737954    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqxm6\" (UniqueName: \"kubernetes.io/projected/66432f27-b3ec-443d-a298-9c1aea8b27f4-kube-api-access-mqxm6\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.838831 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.838794    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-dshm\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.838863    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432f27-b3ec-443d-a298-9c1aea8b27f4-tls-certs\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.838941    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-model-cache\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.838965    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-home\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839223 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.839039    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqxm6\" (UniqueName: \"kubernetes.io/projected/66432f27-b3ec-443d-a298-9c1aea8b27f4-kube-api-access-mqxm6\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839223 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.839093    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-tmp-dir\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.839445    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-model-cache\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839604 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.839579    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-home\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.839740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.839493    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-tmp-dir\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.841879 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.841859    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-dshm\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.842121 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.842102    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432f27-b3ec-443d-a298-9c1aea8b27f4-tls-certs\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.858125 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.858095    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqxm6\" (UniqueName: \"kubernetes.io/projected/66432f27-b3ec-443d-a298-9c1aea8b27f4-kube-api-access-mqxm6\") pod \"sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") " pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:33.959986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:33.959892    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:34.274345 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:12:34.274312    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod66432f27_b3ec_443d_a298_9c1aea8b27f4.slice/crio-813eedcf87676bde0b3eaadf6285dabcab22261a76f3f3b6e9767700694d0b93 WatchSource:0}: Error finding container 813eedcf87676bde0b3eaadf6285dabcab22261a76f3f3b6e9767700694d0b93: Status 404 returned error can't find the container with id 813eedcf87676bde0b3eaadf6285dabcab22261a76f3f3b6e9767700694d0b93
Nov 03 01:12:34.275019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:34.274945    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"]
Nov 03 01:12:34.812732 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:34.812693    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx" event={"ID":"66432f27-b3ec-443d-a298-9c1aea8b27f4","Type":"ContainerStarted","Data":"65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4"}
Nov 03 01:12:34.812732 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:34.812738    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx" event={"ID":"66432f27-b3ec-443d-a298-9c1aea8b27f4","Type":"ContainerStarted","Data":"813eedcf87676bde0b3eaadf6285dabcab22261a76f3f3b6e9767700694d0b93"}
Nov 03 01:12:34.851704 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:34.851642    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx" podStartSLOduration=1.851627173 podStartE2EDuration="1.851627173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:12:34.848569134 +0000 UTC m=+3992.403921159" watchObservedRunningTime="2026-11-03 01:12:34.851627173 +0000 UTC m=+3992.406979198"
Nov 03 01:12:34.960829 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:34.960796    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:35.175795 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:35.175755    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"]
Nov 03 01:12:36.587177 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.587149    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 01:12:36.627615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.627584    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 01:12:36.690327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.690290    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21af1f82-17cb-412b-856e-9f2592be3b4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af1f82-17cb-412b-856e-9f2592be3b4e-plugins-config-volume\") pod \"21af1f82-17cb-412b-856e-9f2592be3b4e\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") "
Nov 03 01:12:36.690534 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.690356    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee39d2b9-9e10-4947-b669-300e04a03415-kube-api-access-8hkqx\" (UniqueName: \"kubernetes.io/projected/ee39d2b9-9e10-4947-b669-300e04a03415-kube-api-access-8hkqx\") pod \"ee39d2b9-9e10-4947-b669-300e04a03415\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") "
Nov 03 01:12:36.690534 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.690459    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee39d2b9-9e10-4947-b669-300e04a03415-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee39d2b9-9e10-4947-b669-300e04a03415-plugins-config-volume\") pod \"ee39d2b9-9e10-4947-b669-300e04a03415\" (UID: \"ee39d2b9-9e10-4947-b669-300e04a03415\") "
Nov 03 01:12:36.690681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.690625    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21af1f82-17cb-412b-856e-9f2592be3b4e-kube-api-access-bvxdw\" (UniqueName: \"kubernetes.io/projected/21af1f82-17cb-412b-856e-9f2592be3b4e-kube-api-access-bvxdw\") pod \"21af1f82-17cb-412b-856e-9f2592be3b4e\" (UID: \"21af1f82-17cb-412b-856e-9f2592be3b4e\") "
Nov 03 01:12:36.690790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.690765    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21af1f82-17cb-412b-856e-9f2592be3b4e-plugins-config-volume" pod "21af1f82-17cb-412b-856e-9f2592be3b4e" (UID: "21af1f82-17cb-412b-856e-9f2592be3b4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:36.690921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.690897    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee39d2b9-9e10-4947-b669-300e04a03415-plugins-config-volume" pod "ee39d2b9-9e10-4947-b669-300e04a03415" (UID: "ee39d2b9-9e10-4947-b669-300e04a03415"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:36.691322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.691285    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee39d2b9-9e10-4947-b669-300e04a03415-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:36.691322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.691313    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21af1f82-17cb-412b-856e-9f2592be3b4e-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:36.692985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.692963    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee39d2b9-9e10-4947-b669-300e04a03415-kube-api-access-8hkqx" pod "ee39d2b9-9e10-4947-b669-300e04a03415" (UID: "ee39d2b9-9e10-4947-b669-300e04a03415"). InnerVolumeSpecName "kube-api-access-8hkqx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:36.693113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.693030    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21af1f82-17cb-412b-856e-9f2592be3b4e-kube-api-access-bvxdw" pod "21af1f82-17cb-412b-856e-9f2592be3b4e" (UID: "21af1f82-17cb-412b-856e-9f2592be3b4e"). InnerVolumeSpecName "kube-api-access-bvxdw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:36.792681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.792590    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvxdw\" (UniqueName: \"kubernetes.io/projected/21af1f82-17cb-412b-856e-9f2592be3b4e-kube-api-access-bvxdw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:36.792681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.792620    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hkqx\" (UniqueName: \"kubernetes.io/projected/ee39d2b9-9e10-4947-b669-300e04a03415-kube-api-access-8hkqx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:36.827867 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.827823    6000 generic.go:356] "Generic (PLEG): container finished" podID="ee39d2b9-9e10-4947-b669-300e04a03415" containerID="50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca" exitCode=0
Nov 03 01:12:36.828052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.827889    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"
Nov 03 01:12:36.828052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.827917    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" event={"ID":"ee39d2b9-9e10-4947-b669-300e04a03415","Type":"ContainerDied","Data":"50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca"}
Nov 03 01:12:36.828052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.827999    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58" event={"ID":"ee39d2b9-9e10-4947-b669-300e04a03415","Type":"ContainerDied","Data":"c54f6ed7af2889bf033a7007a989adfe80864657619782c4425c9336c0cb87d4"}
Nov 03 01:12:36.828052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.828032    6000 scope.go:122] "RemoveContainer" containerID="50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca"
Nov 03 01:12:36.830706 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.830677    6000 generic.go:356] "Generic (PLEG): container finished" podID="21af1f82-17cb-412b-856e-9f2592be3b4e" containerID="9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20" exitCode=0
Nov 03 01:12:36.839572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.831578    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"
Nov 03 01:12:36.839572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.838462    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" event={"ID":"21af1f82-17cb-412b-856e-9f2592be3b4e","Type":"ContainerDied","Data":"9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20"}
Nov 03 01:12:36.839572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.838528    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz" event={"ID":"21af1f82-17cb-412b-856e-9f2592be3b4e","Type":"ContainerDied","Data":"1e4171c01778ef564b047aaca16afefb5fdbed116f74ebe422147a721bde6c78"}
Nov 03 01:12:36.886740 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.886627    6000 scope.go:122] "RemoveContainer" containerID="50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca"
Nov 03 01:12:36.886964 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:36.886941    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca\": container with ID starting with 50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca not found: ID does not exist" containerID="50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca"
Nov 03 01:12:36.887052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.886974    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca"} err="failed to get container status \"50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca\": rpc error: code = NotFound desc = could not find container \"50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca\": container with ID starting with 50f3256cd49920cc35146450bd3251ada05502d407412d4c1fc3bd109122c5ca not found: ID does not exist"
Nov 03 01:12:36.887052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.886996    6000 scope.go:122] "RemoveContainer" containerID="9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20"
Nov 03 01:12:36.898097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.898071    6000 scope.go:122] "RemoveContainer" containerID="9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20"
Nov 03 01:12:36.898464 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:36.898439    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20\": container with ID starting with 9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20 not found: ID does not exist" containerID="9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20"
Nov 03 01:12:36.898556 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.898476    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20"} err="failed to get container status \"9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20\": rpc error: code = NotFound desc = could not find container \"9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20\": container with ID starting with 9dca4f8db96d09fbfcb37c7d42a31ff18c93dd882d927d15acb23ed47880ae20 not found: ID does not exist"
Nov 03 01:12:36.911443 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.911406    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"]
Nov 03 01:12:36.914795 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.914771    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-a07a90-685b6d6d59-gcv58"]
Nov 03 01:12:36.950044 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.950015    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"]
Nov 03 01:12:36.955313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:36.955284    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-a07a90-f77b5cbdc-pskmz"]
Nov 03 01:12:38.567956 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:38.567916    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21af1f82-17cb-412b-856e-9f2592be3b4e" path="/var/lib/kubelet/pods/21af1f82-17cb-412b-856e-9f2592be3b4e/volumes"
Nov 03 01:12:38.568565 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:38.568544    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee39d2b9-9e10-4947-b669-300e04a03415" path="/var/lib/kubelet/pods/ee39d2b9-9e10-4947-b669-300e04a03415/volumes"
Nov 03 01:12:42.542207 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.542171    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"]
Nov 03 01:12:42.542859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.542397    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww" podUID="794f1a8c-6a7c-4289-8907-b9450faa6d04" containerName="discovery" containerID="cri-o://82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11" gracePeriod=30
Nov 03 01:12:42.845364 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.845335    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 01:12:42.874083 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.874051    6000 generic.go:356] "Generic (PLEG): container finished" podID="794f1a8c-6a7c-4289-8907-b9450faa6d04" containerID="82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11" exitCode=0
Nov 03 01:12:42.874315 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.874100    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"
Nov 03 01:12:42.874315 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.874131    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww" event={"ID":"794f1a8c-6a7c-4289-8907-b9450faa6d04","Type":"ContainerDied","Data":"82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11"}
Nov 03 01:12:42.874315 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.874193    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww" event={"ID":"794f1a8c-6a7c-4289-8907-b9450faa6d04","Type":"ContainerDied","Data":"24128938d24d7a9fc3e9aecb63bd4594752579425476320175afe377bc191d13"}
Nov 03 01:12:42.874315 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.874225    6000 scope.go:122] "RemoveContainer" containerID="82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11"
Nov 03 01:12:42.892702 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.892607    6000 scope.go:122] "RemoveContainer" containerID="82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11"
Nov 03 01:12:42.892965 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:42.892934    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11\": container with ID starting with 82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11 not found: ID does not exist" containerID="82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11"
Nov 03 01:12:42.893038 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.892966    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11"} err="failed to get container status \"82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11\": rpc error: code = NotFound desc = could not find container \"82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11\": container with ID starting with 82a231b24ccd8ae6388ce26be0c5f0129552d8fce42b00b10c42300680438b11 not found: ID does not exist"
Nov 03 01:12:42.976322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.976284    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-kube-api-access-9tz8p\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-kube-api-access-9tz8p\") pod \"794f1a8c-6a7c-4289-8907-b9450faa6d04\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") "
Nov 03 01:12:42.976548 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.976344    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-kubeconfig\") pod \"794f1a8c-6a7c-4289-8907-b9450faa6d04\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") "
Nov 03 01:12:42.976548 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.976415    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-dns-cert\") pod \"794f1a8c-6a7c-4289-8907-b9450faa6d04\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") "
Nov 03 01:12:42.976548 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.976539    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/794f1a8c-6a7c-4289-8907-b9450faa6d04-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/794f1a8c-6a7c-4289-8907-b9450faa6d04-local-certs\") pod \"794f1a8c-6a7c-4289-8907-b9450faa6d04\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") "
Nov 03 01:12:42.976734 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.976600    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-ca-configmap\") pod \"794f1a8c-6a7c-4289-8907-b9450faa6d04\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") "
Nov 03 01:12:42.976734 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.976647    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-cacerts\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-cacerts\") pod \"794f1a8c-6a7c-4289-8907-b9450faa6d04\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") "
Nov 03 01:12:42.976844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.976747    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-token\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-token\") pod \"794f1a8c-6a7c-4289-8907-b9450faa6d04\" (UID: \"794f1a8c-6a7c-4289-8907-b9450faa6d04\") "
Nov 03 01:12:42.977450 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.977416    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-ca-configmap" pod "794f1a8c-6a7c-4289-8907-b9450faa6d04" (UID: "794f1a8c-6a7c-4289-8907-b9450faa6d04"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:12:42.978613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.978591    6000 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-ca-configmap\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:42.980105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.980078    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-kube-api-access-9tz8p" pod "794f1a8c-6a7c-4289-8907-b9450faa6d04" (UID: "794f1a8c-6a7c-4289-8907-b9450faa6d04"). InnerVolumeSpecName "kube-api-access-9tz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:42.981187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.981135    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-dns-cert" pod "794f1a8c-6a7c-4289-8907-b9450faa6d04" (UID: "794f1a8c-6a7c-4289-8907-b9450faa6d04"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:42.981323 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.981247    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/794f1a8c-6a7c-4289-8907-b9450faa6d04-local-certs" pod "794f1a8c-6a7c-4289-8907-b9450faa6d04" (UID: "794f1a8c-6a7c-4289-8907-b9450faa6d04"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:42.981323 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.981285    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-token" pod "794f1a8c-6a7c-4289-8907-b9450faa6d04" (UID: "794f1a8c-6a7c-4289-8907-b9450faa6d04"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:42.981461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.981356    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-cacerts" pod "794f1a8c-6a7c-4289-8907-b9450faa6d04" (UID: "794f1a8c-6a7c-4289-8907-b9450faa6d04"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:42.981717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:42.981689    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-kubeconfig" pod "794f1a8c-6a7c-4289-8907-b9450faa6d04" (UID: "794f1a8c-6a7c-4289-8907-b9450faa6d04"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:43.080432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.080357    6000 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-cacerts\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:43.080432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.080382    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:43.080432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.080394    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tz8p\" (UniqueName: \"kubernetes.io/projected/794f1a8c-6a7c-4289-8907-b9450faa6d04-kube-api-access-9tz8p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:43.080432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.080402    6000 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-kubeconfig\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:43.080432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.080412    6000 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/794f1a8c-6a7c-4289-8907-b9450faa6d04-istio-csr-dns-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:43.080432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.080420    6000 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/794f1a8c-6a7c-4289-8907-b9450faa6d04-local-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:43.218616 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.218579    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"]
Nov 03 01:12:43.230663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.230632    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2wkww"]
Nov 03 01:12:43.960377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.960347    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:43.973278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:43.973222    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:44.569007 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:44.568976    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794f1a8c-6a7c-4289-8907-b9450faa6d04" path="/var/lib/kubelet/pods/794f1a8c-6a7c-4289-8907-b9450faa6d04/volumes"
Nov 03 01:12:44.901662 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:44.901635    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:12:49.901996 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:49.901958    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:12:50.199498 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.199404    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm" podUID="5c972578-bd59-4b6b-a4fa-686f8c44d97b" containerName="main" containerID="cri-o://41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144" gracePeriod=45
Nov 03 01:12:50.490059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.490030    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:12:50.680543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680513    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tmp-dir\") pod \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") "
Nov 03 01:12:50.680733 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680602    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-model-cache\") pod \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") "
Nov 03 01:12:50.680733 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680627    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tls-certs\" (UniqueName: \"kubernetes.io/secret/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tls-certs\") pod \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") "
Nov 03 01:12:50.680877 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680769    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-home\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-home\") pod \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") "
Nov 03 01:12:50.680877 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680782    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tmp-dir" pod "5c972578-bd59-4b6b-a4fa-686f8c44d97b" (UID: "5c972578-bd59-4b6b-a4fa-686f8c44d97b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:50.680877 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680810    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c972578-bd59-4b6b-a4fa-686f8c44d97b-kube-api-access-n6xhh\" (UniqueName: \"kubernetes.io/projected/5c972578-bd59-4b6b-a4fa-686f8c44d97b-kube-api-access-n6xhh\") pod \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") "
Nov 03 01:12:50.681018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680876    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-dshm\") pod \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\" (UID: \"5c972578-bd59-4b6b-a4fa-686f8c44d97b\") "
Nov 03 01:12:50.681018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680894    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-model-cache" pod "5c972578-bd59-4b6b-a4fa-686f8c44d97b" (UID: "5c972578-bd59-4b6b-a4fa-686f8c44d97b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:50.681018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.680965    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-home" pod "5c972578-bd59-4b6b-a4fa-686f8c44d97b" (UID: "5c972578-bd59-4b6b-a4fa-686f8c44d97b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:50.681463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.681380    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.681463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.681400    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.681463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.681408    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.683077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.683056    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-dshm" pod "5c972578-bd59-4b6b-a4fa-686f8c44d97b" (UID: "5c972578-bd59-4b6b-a4fa-686f8c44d97b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:12:50.683205 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.683158    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c972578-bd59-4b6b-a4fa-686f8c44d97b-kube-api-access-n6xhh" pod "5c972578-bd59-4b6b-a4fa-686f8c44d97b" (UID: "5c972578-bd59-4b6b-a4fa-686f8c44d97b"). InnerVolumeSpecName "kube-api-access-n6xhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:12:50.683319 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.683212    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tls-certs" pod "5c972578-bd59-4b6b-a4fa-686f8c44d97b" (UID: "5c972578-bd59-4b6b-a4fa-686f8c44d97b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:12:50.782247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.782165    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6xhh\" (UniqueName: \"kubernetes.io/projected/5c972578-bd59-4b6b-a4fa-686f8c44d97b-kube-api-access-n6xhh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.782247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.782191    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5c972578-bd59-4b6b-a4fa-686f8c44d97b-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.782247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.782203    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c972578-bd59-4b6b-a4fa-686f8c44d97b-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:12:50.932541 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.932508    6000 generic.go:356] "Generic (PLEG): container finished" podID="5c972578-bd59-4b6b-a4fa-686f8c44d97b" containerID="41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144" exitCode=0
Nov 03 01:12:50.932993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.932555    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm" event={"ID":"5c972578-bd59-4b6b-a4fa-686f8c44d97b","Type":"ContainerDied","Data":"41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144"}
Nov 03 01:12:50.932993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.932582    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm" event={"ID":"5c972578-bd59-4b6b-a4fa-686f8c44d97b","Type":"ContainerDied","Data":"89f907d7ec3b1bd8ca10441e8b373d26ee9e6c56855465677c20cc381fe8aaf7"}
Nov 03 01:12:50.932993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.932597    6000 scope.go:122] "RemoveContainer" containerID="41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144"
Nov 03 01:12:50.932993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.932557    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"
Nov 03 01:12:50.943634 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.943611    6000 scope.go:122] "RemoveContainer" containerID="41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144"
Nov 03 01:12:50.943896 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:12:50.943876    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144\": container with ID starting with 41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144 not found: ID does not exist" containerID="41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144"
Nov 03 01:12:50.943966 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.943907    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144"} err="failed to get container status \"41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144\": rpc error: code = NotFound desc = could not find container \"41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144\": container with ID starting with 41af562ebe223d42cdb4cbec678f247966f92f6b8f4be3aa440ae0f33a558144 not found: ID does not exist"
Nov 03 01:12:50.982770 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.982740    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"]
Nov 03 01:12:50.985763 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:50.985738    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-065de0/rate-test-model-065de0-kserve-964578485-p6nnm"]
Nov 03 01:12:51.274911 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:51.274867    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:12:51.480415 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:51.480381    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"]
Nov 03 01:12:51.492100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:51.492074    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"]
Nov 03 01:12:52.572189 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:12:52.572157    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c972578-bd59-4b6b-a4fa-686f8c44d97b" path="/var/lib/kubelet/pods/5c972578-bd59-4b6b-a4fa-686f8c44d97b/volumes"
Nov 03 01:13:06.503410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.503340    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main" containerID="cri-o://5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa" gracePeriod=45
Nov 03 01:13:06.513793 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.513726    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main" containerID="cri-o://6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca" gracePeriod=45
Nov 03 01:13:06.802096 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.802069    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 01:13:06.828055 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.828030    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 01:13:06.905460 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905432    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tmp-dir\") pod \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") "
Nov 03 01:13:06.905660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905520    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tls-certs\") pod \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") "
Nov 03 01:13:06.905660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905604    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-tmp-dir\") pod \"65fc04b9-f587-4c16-af73-efb2521cec63\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") "
Nov 03 01:13:06.905660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905652    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-model-cache\") pod \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") "
Nov 03 01:13:06.905847 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905704    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tmp-dir" pod "f877ab10-7fe1-4309-95d4-769dc5b0ae0e" (UID: "f877ab10-7fe1-4309-95d4-769dc5b0ae0e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.905847 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905738    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-dshm\") pod \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") "
Nov 03 01:13:06.905847 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905766    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65fc04b9-f587-4c16-af73-efb2521cec63-tls-certs\" (UniqueName: \"kubernetes.io/secret/65fc04b9-f587-4c16-af73-efb2521cec63-tls-certs\") pod \"65fc04b9-f587-4c16-af73-efb2521cec63\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") "
Nov 03 01:13:06.905847 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905816    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-dshm\") pod \"65fc04b9-f587-4c16-af73-efb2521cec63\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") "
Nov 03 01:13:06.906058 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905874    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-kube-api-access-rvrth\" (UniqueName: \"kubernetes.io/projected/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-kube-api-access-rvrth\") pod \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") "
Nov 03 01:13:06.906058 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.905997    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65fc04b9-f587-4c16-af73-efb2521cec63-kube-api-access-khhc9\" (UniqueName: \"kubernetes.io/projected/65fc04b9-f587-4c16-af73-efb2521cec63-kube-api-access-khhc9\") pod \"65fc04b9-f587-4c16-af73-efb2521cec63\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") "
Nov 03 01:13:06.906058 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.906028    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-home\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-home\") pod \"65fc04b9-f587-4c16-af73-efb2521cec63\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") "
Nov 03 01:13:06.906226 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.906068    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-model-cache\") pod \"65fc04b9-f587-4c16-af73-efb2521cec63\" (UID: \"65fc04b9-f587-4c16-af73-efb2521cec63\") "
Nov 03 01:13:06.906226 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.906109    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-home\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-home\") pod \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\" (UID: \"f877ab10-7fe1-4309-95d4-769dc5b0ae0e\") "
Nov 03 01:13:06.906765 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.906740    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:06.906993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.906975    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-home" pod "f877ab10-7fe1-4309-95d4-769dc5b0ae0e" (UID: "f877ab10-7fe1-4309-95d4-769dc5b0ae0e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.907084 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.906997    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-model-cache" pod "f877ab10-7fe1-4309-95d4-769dc5b0ae0e" (UID: "f877ab10-7fe1-4309-95d4-769dc5b0ae0e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.907196 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.907180    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-home" pod "65fc04b9-f587-4c16-af73-efb2521cec63" (UID: "65fc04b9-f587-4c16-af73-efb2521cec63"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.907491 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.907415    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-model-cache" pod "65fc04b9-f587-4c16-af73-efb2521cec63" (UID: "65fc04b9-f587-4c16-af73-efb2521cec63"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.908330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.908305    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-tmp-dir" pod "65fc04b9-f587-4c16-af73-efb2521cec63" (UID: "65fc04b9-f587-4c16-af73-efb2521cec63"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.908733 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.908694    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-dshm" pod "65fc04b9-f587-4c16-af73-efb2521cec63" (UID: "65fc04b9-f587-4c16-af73-efb2521cec63"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.909108 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.909057    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tls-certs" pod "f877ab10-7fe1-4309-95d4-769dc5b0ae0e" (UID: "f877ab10-7fe1-4309-95d4-769dc5b0ae0e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:06.909609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.909579    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65fc04b9-f587-4c16-af73-efb2521cec63-kube-api-access-khhc9" pod "65fc04b9-f587-4c16-af73-efb2521cec63" (UID: "65fc04b9-f587-4c16-af73-efb2521cec63"). InnerVolumeSpecName "kube-api-access-khhc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:06.910041 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.910019    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-dshm" pod "f877ab10-7fe1-4309-95d4-769dc5b0ae0e" (UID: "f877ab10-7fe1-4309-95d4-769dc5b0ae0e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:06.910433 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.910406    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-kube-api-access-rvrth" pod "f877ab10-7fe1-4309-95d4-769dc5b0ae0e" (UID: "f877ab10-7fe1-4309-95d4-769dc5b0ae0e"). InnerVolumeSpecName "kube-api-access-rvrth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:06.910800 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:06.910782    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65fc04b9-f587-4c16-af73-efb2521cec63-tls-certs" pod "65fc04b9-f587-4c16-af73-efb2521cec63" (UID: "65fc04b9-f587-4c16-af73-efb2521cec63"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:07.008059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008023    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008054    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/65fc04b9-f587-4c16-af73-efb2521cec63-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008063    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008071    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvrth\" (UniqueName: \"kubernetes.io/projected/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-kube-api-access-rvrth\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008080    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhc9\" (UniqueName: \"kubernetes.io/projected/65fc04b9-f587-4c16-af73-efb2521cec63-kube-api-access-khhc9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008088    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008096    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008103    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008111    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008119    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/65fc04b9-f587-4c16-af73-efb2521cec63-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.008360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.008126    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f877ab10-7fe1-4309-95d4-769dc5b0ae0e-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:07.045876 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.045792    6000 generic.go:356] "Generic (PLEG): container finished" podID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerID="6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca" exitCode=0
Nov 03 01:13:07.045876 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.045838    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"
Nov 03 01:13:07.046097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.045885    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" event={"ID":"f877ab10-7fe1-4309-95d4-769dc5b0ae0e","Type":"ContainerDied","Data":"6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca"}
Nov 03 01:13:07.046097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.045928    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6" event={"ID":"f877ab10-7fe1-4309-95d4-769dc5b0ae0e","Type":"ContainerDied","Data":"79cbf0bdc64bb04e23b572cdc2d3d1ba3d33841950c67bf0ce2ce2c41ce9e558"}
Nov 03 01:13:07.046097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.045955    6000 scope.go:122] "RemoveContainer" containerID="6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca"
Nov 03 01:13:07.047611 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.047572    6000 generic.go:356] "Generic (PLEG): container finished" podID="65fc04b9-f587-4c16-af73-efb2521cec63" containerID="5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa" exitCode=0
Nov 03 01:13:07.047753 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.047618    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" event={"ID":"65fc04b9-f587-4c16-af73-efb2521cec63","Type":"ContainerDied","Data":"5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa"}
Nov 03 01:13:07.047753 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.047656    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr" event={"ID":"65fc04b9-f587-4c16-af73-efb2521cec63","Type":"ContainerDied","Data":"f8aa96241fcba8c1a32269bd8c06aa30a8927bf8a6e345265fd4c092db989f15"}
Nov 03 01:13:07.047753 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.047624    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"
Nov 03 01:13:07.057639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.057623    6000 scope.go:122] "RemoveContainer" containerID="6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca"
Nov 03 01:13:07.057946 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:13:07.057881    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca\": container with ID starting with 6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca not found: ID does not exist" containerID="6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca"
Nov 03 01:13:07.057946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.057912    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca"} err="failed to get container status \"6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca\": rpc error: code = NotFound desc = could not find container \"6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca\": container with ID starting with 6d02818d2f03981c057a7bb5d3a1bb35bd04f046d07d37528d37497a410ffdca not found: ID does not exist"
Nov 03 01:13:07.057946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.057933    6000 scope.go:122] "RemoveContainer" containerID="5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa"
Nov 03 01:13:07.067307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.067289    6000 scope.go:122] "RemoveContainer" containerID="5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa"
Nov 03 01:13:07.067553 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:13:07.067532    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa\": container with ID starting with 5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa not found: ID does not exist" containerID="5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa"
Nov 03 01:13:07.067612 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.067559    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa"} err="failed to get container status \"5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa\": rpc error: code = NotFound desc = could not find container \"5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa\": container with ID starting with 5c7f27b91c277924368277c381d0522c44a0f18605bb0a0d661a55dd39da29fa not found: ID does not exist"
Nov 03 01:13:07.093832 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.093809    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"]
Nov 03 01:13:07.099352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.099331    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/premium-w0-fc9316-kserve-694779777d-9qlr6"]
Nov 03 01:13:07.131348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.131325    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"]
Nov 03 01:13:07.134194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:07.134175    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-fc9316/distinct-w0-fc9316-kserve-7c8f55bd5c-qq4fr"]
Nov 03 01:13:08.567918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:08.567881    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" path="/var/lib/kubelet/pods/65fc04b9-f587-4c16-af73-efb2521cec63/volumes"
Nov 03 01:13:08.568433 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:08.568417    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" path="/var/lib/kubelet/pods/f877ab10-7fe1-4309-95d4-769dc5b0ae0e/volumes"
Nov 03 01:13:19.898359 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:13:19.898294    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:13:27.759432 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.759398    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"]
Nov 03 01:13:27.760067 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760038    6000 cpu_manager.go:414] "containerMap: removing container" podUID="5c972578-bd59-4b6b-a4fa-686f8c44d97b" containerName="main"
Nov 03 01:13:27.760067 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760061    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c972578-bd59-4b6b-a4fa-686f8c44d97b" containerName="main"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760085    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760092    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760121    6000 cpu_manager.go:414] "containerMap: removing container" podUID="794f1a8c-6a7c-4289-8907-b9450faa6d04" containerName="discovery"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760129    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794f1a8c-6a7c-4289-8907-b9450faa6d04" containerName="discovery"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760163    6000 cpu_manager.go:414] "containerMap: removing container" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760170    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760185    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ee39d2b9-9e10-4947-b669-300e04a03415" containerName="payload-processing"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760192    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee39d2b9-9e10-4947-b669-300e04a03415" containerName="payload-processing"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760225    6000 cpu_manager.go:414] "containerMap: removing container" podUID="21af1f82-17cb-412b-856e-9f2592be3b4e" containerName="payload-pre-processing"
Nov 03 01:13:27.760317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760233    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21af1f82-17cb-412b-856e-9f2592be3b4e" containerName="payload-pre-processing"
Nov 03 01:13:27.760724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760400    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="794f1a8c-6a7c-4289-8907-b9450faa6d04" containerName="discovery"
Nov 03 01:13:27.760724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760418    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="65fc04b9-f587-4c16-af73-efb2521cec63" containerName="main"
Nov 03 01:13:27.760724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760428    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee39d2b9-9e10-4947-b669-300e04a03415" containerName="payload-processing"
Nov 03 01:13:27.760724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760445    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="21af1f82-17cb-412b-856e-9f2592be3b4e" containerName="payload-pre-processing"
Nov 03 01:13:27.760724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760458    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c972578-bd59-4b6b-a4fa-686f8c44d97b" containerName="main"
Nov 03 01:13:27.760724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.760470    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f877ab10-7fe1-4309-95d4-769dc5b0ae0e" containerName="main"
Nov 03 01:13:27.765438 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.765420    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.783514 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.783481    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-fc9316-openshift-default-dockercfg-8zk4q\""
Nov 03 01:13:27.798014 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.797980    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"]
Nov 03 01:13:27.873397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873362    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.873397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873401    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.873615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873418    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zcvp\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-kube-api-access-8zcvp\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.873615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873499    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.873697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873620    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.873733 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873684    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.873772 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873759    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.873942 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.873926    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.874023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.874009    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975378 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975342    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975605 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975413    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975605 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975456    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975605 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975479    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975605 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975501    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zcvp\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-kube-api-access-8zcvp\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975646    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975745    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975787    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975830    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.975850 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975831    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-credential-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.976143 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975883    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-data\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.976143 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.975917    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-certs\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.976143 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.976090    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-socket\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.976928 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.976903    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istiod-ca-cert\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.977963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.977937    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-envoy\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.978196 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.978178    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-podinfo\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.995367 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.995340    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-token\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:27.996008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:27.995978    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zcvp\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-kube-api-access-8zcvp\") pod \"e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") " pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:28.079313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:28.079212    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:28.414582 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:28.414553    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"]
Nov 03 01:13:28.416350 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:13:28.416323    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d63df94_9b02_4c1b_aae4_1e52e57b5d9e.slice/crio-42405d986b42a66a74b5af3449b0041eb9ee7da92b3875ee490c9ea6858b0180 WatchSource:0}: Error finding container 42405d986b42a66a74b5af3449b0041eb9ee7da92b3875ee490c9ea6858b0180: Status 404 returned error can't find the container with id 42405d986b42a66a74b5af3449b0041eb9ee7da92b3875ee490c9ea6858b0180
Nov 03 01:13:28.418956 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:28.418924    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:13:28.419019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:28.418990    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:13:28.419057 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:28.419016    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:13:29.198859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:29.198823    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" event={"ID":"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e","Type":"ContainerStarted","Data":"c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f"}
Nov 03 01:13:29.199381 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:29.198886    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" event={"ID":"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e","Type":"ContainerStarted","Data":"42405d986b42a66a74b5af3449b0041eb9ee7da92b3875ee490c9ea6858b0180"}
Nov 03 01:13:29.243419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:29.243368    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podStartSLOduration=2.243353516 podStartE2EDuration="2.243353516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:13:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:13:29.241503956 +0000 UTC m=+4046.796855983" watchObservedRunningTime="2026-11-03 01:13:29.243353516 +0000 UTC m=+4046.798705540"
Nov 03 01:13:30.080306 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:30.080241    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:30.082311 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:30.082286    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 01:13:30.082409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:30.082340    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 01:13:31.079803 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:31.079772    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 01:13:31.080238 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:31.079830    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 01:13:32.079835 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:32.079801    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 01:13:32.080315 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:32.079852    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 01:13:33.080552 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:33.080512    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 01:13:33.081033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:33.080579    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 01:13:34.079942 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:34.079900    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 01:13:34.080140 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:34.079957    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 01:13:35.080203 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:35.080161    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 01:13:35.080666 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:35.080237    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 01:13:35.698411 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:35.698376    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:13:36.080200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:36.080114    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused" start-of-body=
Nov 03 01:13:36.080403 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:36.080199    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.37:15021/healthz/ready\": dial tcp 10.134.1.37:15021: connect: connection refused"
Nov 03 01:13:38.014633 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:38.014596    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:38.015288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:38.015234    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:38.028624 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:38.028589    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:43.933495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:43.933421    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"]
Nov 03 01:13:43.933943 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:43.933641    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6" podUID="c7c6efed-66cc-43c5-b23c-2b09f33508a5" containerName="maas-api" containerID="cri-o://82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c" gracePeriod=30
Nov 03 01:13:44.212114 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.212080    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 01:13:44.300328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.300287    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-metrics-certs\") pod \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") "
Nov 03 01:13:44.300502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.300411    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7c6efed-66cc-43c5-b23c-2b09f33508a5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c6efed-66cc-43c5-b23c-2b09f33508a5-openshift-service-ca\") pod \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") "
Nov 03 01:13:44.300502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.300456    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c6efed-66cc-43c5-b23c-2b09f33508a5-kube-api-access-js8qq\" (UniqueName: \"kubernetes.io/projected/c7c6efed-66cc-43c5-b23c-2b09f33508a5-kube-api-access-js8qq\") pod \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") "
Nov 03 01:13:44.300583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.300504    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-maas-api-tls\") pod \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\" (UID: \"c7c6efed-66cc-43c5-b23c-2b09f33508a5\") "
Nov 03 01:13:44.300859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.300822    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7c6efed-66cc-43c5-b23c-2b09f33508a5-openshift-service-ca" pod "c7c6efed-66cc-43c5-b23c-2b09f33508a5" (UID: "c7c6efed-66cc-43c5-b23c-2b09f33508a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:44.301064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.301046    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c7c6efed-66cc-43c5-b23c-2b09f33508a5-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:44.302716 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.302687    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-maas-api-tls" pod "c7c6efed-66cc-43c5-b23c-2b09f33508a5" (UID: "c7c6efed-66cc-43c5-b23c-2b09f33508a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:44.302822 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.302716    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-metrics-certs" pod "c7c6efed-66cc-43c5-b23c-2b09f33508a5" (UID: "c7c6efed-66cc-43c5-b23c-2b09f33508a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:13:44.302908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.302889    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c6efed-66cc-43c5-b23c-2b09f33508a5-kube-api-access-js8qq" pod "c7c6efed-66cc-43c5-b23c-2b09f33508a5" (UID: "c7c6efed-66cc-43c5-b23c-2b09f33508a5"). InnerVolumeSpecName "kube-api-access-js8qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:44.325513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.325483    6000 generic.go:356] "Generic (PLEG): container finished" podID="c7c6efed-66cc-43c5-b23c-2b09f33508a5" containerID="82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c" exitCode=0
Nov 03 01:13:44.325678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.325543    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"
Nov 03 01:13:44.325678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.325561    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6" event={"ID":"c7c6efed-66cc-43c5-b23c-2b09f33508a5","Type":"ContainerDied","Data":"82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c"}
Nov 03 01:13:44.325678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.325623    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6" event={"ID":"c7c6efed-66cc-43c5-b23c-2b09f33508a5","Type":"ContainerDied","Data":"1425bcbe93e4df752524e38637426e409065cf5a25df74dd7fcb4639be0bd01a"}
Nov 03 01:13:44.325678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.325653    6000 scope.go:122] "RemoveContainer" containerID="82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c"
Nov 03 01:13:44.340417 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.340396    6000 scope.go:122] "RemoveContainer" containerID="82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c"
Nov 03 01:13:44.340730 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:13:44.340709    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c\": container with ID starting with 82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c not found: ID does not exist" containerID="82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c"
Nov 03 01:13:44.340790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.340742    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c"} err="failed to get container status \"82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c\": rpc error: code = NotFound desc = could not find container \"82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c\": container with ID starting with 82fd741f0265882a52b12dac0b5d6ca2ee2623a992952b04b2f35a6a3b92e30c not found: ID does not exist"
Nov 03 01:13:44.374645 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.374611    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"]
Nov 03 01:13:44.379008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.378982    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-fc9316-5f59486575-mzbt6"]
Nov 03 01:13:44.404421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.404340    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:44.404421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.404381    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-js8qq\" (UniqueName: \"kubernetes.io/projected/c7c6efed-66cc-43c5-b23c-2b09f33508a5-kube-api-access-js8qq\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:44.404421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.404395    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c7c6efed-66cc-43c5-b23c-2b09f33508a5-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:44.569713 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:44.569630    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c6efed-66cc-43c5-b23c-2b09f33508a5" path="/var/lib/kubelet/pods/c7c6efed-66cc-43c5-b23c-2b09f33508a5/volumes"
Nov 03 01:13:49.897194 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:13:49.897155    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:13:51.585711 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:51.585678    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"]
Nov 03 01:13:51.586503 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:51.586470    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" containerID="cri-o://c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f" gracePeriod=30
Nov 03 01:13:53.017514 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:53.017470    6000 patch_prober.go:29] interesting pod/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:13:53.017942 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:53.017537    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:13:56.868798 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.868717    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:56.986518 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986485    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-credential-socket\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.986720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986556    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istiod-ca-cert\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.986720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986606    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-data\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.986720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986649    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-kube-api-access-8zcvp\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-kube-api-access-8zcvp\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.986720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986682    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-envoy\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.986962 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986747    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-credential-socket" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.986962 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986767    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-podinfo\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.986962 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986812    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-certs\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.986962 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986933    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-token\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-token\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.987176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.986958    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-data" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.987176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987004    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istiod-ca-cert" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:13:56.987176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987010    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-socket\") pod \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\" (UID: \"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e\") "
Nov 03 01:13:56.987419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987187    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-certs" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.987419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987249    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-socket" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.987853 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987829    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.987905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987861    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.987905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987877    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.987905 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987892    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.988002 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.987905    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:56.989125 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.989098    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-envoy" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:13:56.989300 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.989277    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-token" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:56.989600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.989568    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-kube-api-access-8zcvp" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "kube-api-access-8zcvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:13:56.989600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:56.989575    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-podinfo" pod "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" (UID: "1d63df94-9b02-4c1b-aae4-1e52e57b5d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:13:57.088846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.088812    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zcvp\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-kube-api-access-8zcvp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.088846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.088843    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.089059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.088858    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.089059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.088869    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:13:57.406887 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.406838    6000 generic.go:356] "Generic (PLEG): container finished" podID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerID="c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f" exitCode=0
Nov 03 01:13:57.407079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.406918    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"
Nov 03 01:13:57.407079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.406918    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" event={"ID":"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e","Type":"ContainerDied","Data":"c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f"}
Nov 03 01:13:57.407079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.407014    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts" event={"ID":"1d63df94-9b02-4c1b-aae4-1e52e57b5d9e","Type":"ContainerDied","Data":"42405d986b42a66a74b5af3449b0041eb9ee7da92b3875ee490c9ea6858b0180"}
Nov 03 01:13:57.407079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.407044    6000 scope.go:122] "RemoveContainer" containerID="c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f"
Nov 03 01:13:57.418816 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.418794    6000 scope.go:122] "RemoveContainer" containerID="c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f"
Nov 03 01:13:57.419118 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:13:57.419096    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f\": container with ID starting with c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f not found: ID does not exist" containerID="c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f"
Nov 03 01:13:57.419191 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.419131    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f"} err="failed to get container status \"c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f\": rpc error: code = NotFound desc = could not find container \"c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f\": container with ID starting with c7d4df88b3ea92b320691388b9215efd293640577e6d247e838bc89ab70d682f not found: ID does not exist"
Nov 03 01:13:57.458457 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.458430    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"]
Nov 03 01:13:57.470107 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:57.470083    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-fc9316-openshift-default-58b7b57d89-vmsts"]
Nov 03 01:13:58.568763 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:13:58.568729    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" path="/var/lib/kubelet/pods/1d63df94-9b02-4c1b-aae4-1e52e57b5d9e/volumes"
Nov 03 01:14:01.210314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:01.210269    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"]
Nov 03 01:14:01.247908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:01.247874    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"]
Nov 03 01:14:06.211571 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:06.211515    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerName="payload-processing" containerID="cri-o://7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb" gracePeriod=25
Nov 03 01:14:06.248327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:06.248268    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" containerName="payload-pre-processing" containerID="cri-o://5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f" gracePeriod=25
Nov 03 01:14:06.334097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:06.334057    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:06.334296 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:06.334097    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:06.747677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:06.747645    6000 scope.go:122] "RemoveContainer" containerID="60377d499bf149c9a8f20caff2df16103a6ec643145663af57d7b2a0d47832f0"
Nov 03 01:14:12.640217 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.640185    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"]
Nov 03 01:14:12.640861 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.640443    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy" containerID="cri-o://afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a" gracePeriod=30
Nov 03 01:14:12.651717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.651690    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"]
Nov 03 01:14:12.651961 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.651939    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy" containerID="cri-o://9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3" gracePeriod=30
Nov 03 01:14:12.654050 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.654024    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"]
Nov 03 01:14:12.654211 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.654192    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy" containerID="cri-o://0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5" gracePeriod=30
Nov 03 01:14:12.663215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.663190    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"]
Nov 03 01:14:12.663436 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:12.663414    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy" containerID="cri-o://60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4" gracePeriod=30
Nov 03 01:14:16.334700 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:16.334659    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:16.335147 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:16.334659    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:14:17.938191 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:17.937810    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 01:14:17.970415 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:17.970387    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 01:14:18.050830 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.050805    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-socket\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.050847    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-istio-token\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-istio-token\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.050890    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d54cd764-315b-4333-960e-1b442ed0ab93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54cd764-315b-4333-960e-1b442ed0ab93-istiod-ca-cert\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.050922    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-data\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.050970    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-certs\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051011    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d54cd764-315b-4333-960e-1b442ed0ab93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54cd764-315b-4333-960e-1b442ed0ab93-istio-podinfo\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051063    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-token\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051522 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051106    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-data\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051522 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051198    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-certs\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051522 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051325    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-podinfo\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051522 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051378    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istiod-ca-cert\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051522 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051416    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-envoy\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051522 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051442    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-kube-api-access-h878x\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-kube-api-access-h878x\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051517    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-credential-socket\") pod \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\" (UID: \"0a1e7b0f-2b36-4171-8233-395051e2fb8d\") "
Nov 03 01:14:18.051865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051574    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-envoy\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051605    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-credential-socket\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051642    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-socket\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.051865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.051698    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-kube-api-access-djz2r\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-kube-api-access-djz2r\") pod \"d54cd764-315b-4333-960e-1b442ed0ab93\" (UID: \"d54cd764-315b-4333-960e-1b442ed0ab93\") "
Nov 03 01:14:18.052664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.052626    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-socket" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.053760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.053737    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-credential-socket" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.054954 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.054916    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-socket" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.055413 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.055121    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-data" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.056450 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.056422    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-certs" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.056564 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.056494    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-istio-token" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.056643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.056581    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-data" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.056840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.056816    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-podinfo" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:14:18.056918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.056876    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-certs" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.056980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.056915    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d54cd764-315b-4333-960e-1b442ed0ab93-istiod-ca-cert" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:18.057296 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.055065    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-credential-socket" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.057587 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.057564    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istiod-ca-cert" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:18.059187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.059161    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-kube-api-access-h878x" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "kube-api-access-h878x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.059517 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.059403    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-envoy" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.059517 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.059415    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d54cd764-315b-4333-960e-1b442ed0ab93-istio-podinfo" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:14:18.059517 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.059454    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-kube-api-access-djz2r" pod "d54cd764-315b-4333-960e-1b442ed0ab93" (UID: "d54cd764-315b-4333-960e-1b442ed0ab93"). InnerVolumeSpecName "kube-api-access-djz2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.059517 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.059491    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-token" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.060504 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.060480    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-envoy" pod "0a1e7b0f-2b36-4171-8233-395051e2fb8d" (UID: "0a1e7b0f-2b36-4171-8233-395051e2fb8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.061479 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.061456    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 01:14:18.063616 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.063596    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:14:18.153489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153455    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-data\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.153712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153530    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-envoy\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.153712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153563    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-podinfo\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.153712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153590    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istiod-ca-cert\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.153712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153638    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-token\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.153712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153667    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-token\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-token\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.154037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153733    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-kube-api-access-d2kg6\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-kube-api-access-d2kg6\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.154037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153798    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-credential-socket\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.154037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153835    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b5e4b26-c21e-4fde-b787-33bb4685879b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e4b26-c21e-4fde-b787-33bb4685879b-istiod-ca-cert\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.154037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153887    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-podinfo\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.154037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153913    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-socket\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.154037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.153987    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-envoy\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.154037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154018    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-credential-socket\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.154466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154077    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-socket\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.154466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154108    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-data\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.154466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154167    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-kube-api-access-xvv9t\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-kube-api-access-xvv9t\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.154466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154209    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-certs\") pod \"2b5e4b26-c21e-4fde-b787-33bb4685879b\" (UID: \"2b5e4b26-c21e-4fde-b787-33bb4685879b\") "
Nov 03 01:14:18.154466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154214    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-credential-socket" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.154466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154280    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-certs\") pod \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\" (UID: \"82f4fb1a-2c68-49ce-9cc6-8c18e4949548\") "
Nov 03 01:14:18.154818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154790    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istiod-ca-cert" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:18.154910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154886    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.154981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154915    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.154981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154931    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d54cd764-315b-4333-960e-1b442ed0ab93-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.154981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154945    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.154981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154957    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.154981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154968    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.154981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154981    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d54cd764-315b-4333-960e-1b442ed0ab93-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.154995    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155007    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155019    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155031    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155043    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155055    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155068    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h878x\" (UniqueName: \"kubernetes.io/projected/0a1e7b0f-2b36-4171-8233-395051e2fb8d-kube-api-access-h878x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155081    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a1e7b0f-2b36-4171-8233-395051e2fb8d-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155095    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155108    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155122    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d54cd764-315b-4333-960e-1b442ed0ab93-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155135    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djz2r\" (UniqueName: \"kubernetes.io/projected/d54cd764-315b-4333-960e-1b442ed0ab93-kube-api-access-djz2r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155038    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-credential-socket" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155370    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b5e4b26-c21e-4fde-b787-33bb4685879b-istiod-ca-cert" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:18.155385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155373    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-socket" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.156231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155545    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-data" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.156231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155734    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-data" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.156231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.155924    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-socket" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.156231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.156115    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-certs" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.156231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.156122    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-certs" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.159201 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.159174    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-kube-api-access-d2kg6" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "kube-api-access-d2kg6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.160057 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.160018    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-envoy" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.160166 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.160125    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-kube-api-access-xvv9t" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "kube-api-access-xvv9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.160629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.160605    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-token" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.160745 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.160642    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-token" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:18.160745 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.160709    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-envoy" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:18.160900 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.160790    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-podinfo" pod "82f4fb1a-2c68-49ce-9cc6-8c18e4949548" (UID: "82f4fb1a-2c68-49ce-9cc6-8c18e4949548"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:14:18.162079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.162055    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-podinfo" pod "2b5e4b26-c21e-4fde-b787-33bb4685879b" (UID: "2b5e4b26-c21e-4fde-b787-33bb4685879b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:14:18.256174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256089    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256122    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256134    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256147    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256160    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256170    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256174 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256179    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256188    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2kg6\" (UniqueName: \"kubernetes.io/projected/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-kube-api-access-d2kg6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256199    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b5e4b26-c21e-4fde-b787-33bb4685879b-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256206    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b5e4b26-c21e-4fde-b787-33bb4685879b-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256217    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256228    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256236    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256245    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256284    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82f4fb1a-2c68-49ce-9cc6-8c18e4949548-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256293    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xvv9t\" (UniqueName: \"kubernetes.io/projected/2b5e4b26-c21e-4fde-b787-33bb4685879b-kube-api-access-xvv9t\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.256546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.256301    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b5e4b26-c21e-4fde-b787-33bb4685879b-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:18.540301 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.540182    6000 generic.go:356] "Generic (PLEG): container finished" podID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerID="0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5" exitCode=0
Nov 03 01:14:18.540301 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.540234    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"
Nov 03 01:14:18.540567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.540291    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" event={"ID":"2b5e4b26-c21e-4fde-b787-33bb4685879b","Type":"ContainerDied","Data":"0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5"}
Nov 03 01:14:18.540567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.540350    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4" event={"ID":"2b5e4b26-c21e-4fde-b787-33bb4685879b","Type":"ContainerDied","Data":"0cdc0918152a6e75926d890bca39a2bd241c914b5ce32a8a022bf28d8590d638"}
Nov 03 01:14:18.540567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.540379    6000 scope.go:122] "RemoveContainer" containerID="0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5"
Nov 03 01:14:18.541873 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.541850    6000 generic.go:356] "Generic (PLEG): container finished" podID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerID="60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4" exitCode=0
Nov 03 01:14:18.541986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.541888    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"
Nov 03 01:14:18.541986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.541953    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" event={"ID":"82f4fb1a-2c68-49ce-9cc6-8c18e4949548","Type":"ContainerDied","Data":"60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4"}
Nov 03 01:14:18.542086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.542012    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94" event={"ID":"82f4fb1a-2c68-49ce-9cc6-8c18e4949548","Type":"ContainerDied","Data":"7030151cf4ba1ed22330136dc8a889f559095ca7fcb20652e53059cc5de12b05"}
Nov 03 01:14:18.547397 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.547374    6000 generic.go:356] "Generic (PLEG): container finished" podID="d54cd764-315b-4333-960e-1b442ed0ab93" containerID="9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3" exitCode=0
Nov 03 01:14:18.547578 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.547415    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"
Nov 03 01:14:18.547578 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.547459    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" event={"ID":"d54cd764-315b-4333-960e-1b442ed0ab93","Type":"ContainerDied","Data":"9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3"}
Nov 03 01:14:18.547578 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.547534    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx" event={"ID":"d54cd764-315b-4333-960e-1b442ed0ab93","Type":"ContainerDied","Data":"40aa4169fceff3d3bf846484883a3ff9d94e20844da73e0d967298f8a45793c2"}
Nov 03 01:14:18.548860 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.548837    6000 generic.go:356] "Generic (PLEG): container finished" podID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerID="afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a" exitCode=0
Nov 03 01:14:18.548950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.548878    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" event={"ID":"0a1e7b0f-2b36-4171-8233-395051e2fb8d","Type":"ContainerDied","Data":"afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a"}
Nov 03 01:14:18.548950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.548914    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48" event={"ID":"0a1e7b0f-2b36-4171-8233-395051e2fb8d","Type":"ContainerDied","Data":"77820c775d0023d6746aaedcbb11d71162afb5f28515918566cd66ccc6591366"}
Nov 03 01:14:18.549030 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.548986    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"
Nov 03 01:14:18.553042 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.553024    6000 scope.go:122] "RemoveContainer" containerID="0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5"
Nov 03 01:14:18.553410 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:18.553390    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5\": container with ID starting with 0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5 not found: ID does not exist" containerID="0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5"
Nov 03 01:14:18.553513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.553417    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5"} err="failed to get container status \"0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5\": rpc error: code = NotFound desc = could not find container \"0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5\": container with ID starting with 0f936fed721431645d75ced961623004aec2adf1d592713ebddd10f46bbf10e5 not found: ID does not exist"
Nov 03 01:14:18.553513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.553439    6000 scope.go:122] "RemoveContainer" containerID="60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4"
Nov 03 01:14:18.566680 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.566661    6000 scope.go:122] "RemoveContainer" containerID="60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4"
Nov 03 01:14:18.566982 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:18.566957    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4\": container with ID starting with 60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4 not found: ID does not exist" containerID="60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4"
Nov 03 01:14:18.567084 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.566995    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4"} err="failed to get container status \"60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4\": rpc error: code = NotFound desc = could not find container \"60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4\": container with ID starting with 60fb5abd1af1ac990e3091d7db606cff57d02a993f9b483fff4786cbb05f51d4 not found: ID does not exist"
Nov 03 01:14:18.567084 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.567015    6000 scope.go:122] "RemoveContainer" containerID="9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3"
Nov 03 01:14:18.577322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.577301    6000 scope.go:122] "RemoveContainer" containerID="9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3"
Nov 03 01:14:18.577592 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:18.577573    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3\": container with ID starting with 9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3 not found: ID does not exist" containerID="9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3"
Nov 03 01:14:18.577658 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.577599    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3"} err="failed to get container status \"9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3\": rpc error: code = NotFound desc = could not find container \"9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3\": container with ID starting with 9854cfa58290ad1dac975c40ca609f99b4690c99870daafdc5d8e8cd7c7a9de3 not found: ID does not exist"
Nov 03 01:14:18.577658 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.577616    6000 scope.go:122] "RemoveContainer" containerID="afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a"
Nov 03 01:14:18.586660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.586643    6000 scope.go:122] "RemoveContainer" containerID="afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a"
Nov 03 01:14:18.586941 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:18.586925    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a\": container with ID starting with afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a not found: ID does not exist" containerID="afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a"
Nov 03 01:14:18.586995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.586949    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a"} err="failed to get container status \"afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a\": rpc error: code = NotFound desc = could not find container \"afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a\": container with ID starting with afd32362f2fc88871c52a2ee644069ead47f62b5742ffe65120e08433359829a not found: ID does not exist"
Nov 03 01:14:18.633573 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.633547    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"]
Nov 03 01:14:18.644724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.644701    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-666b4"]
Nov 03 01:14:18.671552 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.671517    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"]
Nov 03 01:14:18.674840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.674700    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dqqvx"]
Nov 03 01:14:18.706708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.706684    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"]
Nov 03 01:14:18.713414 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.713392    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-zqd94"]
Nov 03 01:14:18.745294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.745239    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"]
Nov 03 01:14:18.747894 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:18.747870    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w5m48"]
Nov 03 01:14:19.900669 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:19.900633    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:14:20.567016 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:20.566986    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" path="/var/lib/kubelet/pods/0a1e7b0f-2b36-4171-8233-395051e2fb8d/volumes"
Nov 03 01:14:20.567502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:20.567487    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" path="/var/lib/kubelet/pods/2b5e4b26-c21e-4fde-b787-33bb4685879b/volumes"
Nov 03 01:14:20.567899 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:20.567886    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" path="/var/lib/kubelet/pods/82f4fb1a-2c68-49ce-9cc6-8c18e4949548/volumes"
Nov 03 01:14:20.568656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:20.568638    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" path="/var/lib/kubelet/pods/d54cd764-315b-4333-960e-1b442ed0ab93/volumes"
Nov 03 01:14:22.436845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:22.436810    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:14:24.211314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:24.211284    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"]
Nov 03 01:14:24.213587 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:24.211562    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy" containerID="cri-o://8e20ed78bfd07682ebe0fb77fb068e5b2b007261dc6a83651d51c38475add2ca" gracePeriod=30
Nov 03 01:14:26.333735 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:14:26.333706    6000 logging.go:55] [core] [Channel #19964 SubChannel #19966]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.147:50052", ServerName: "10.134.0.147:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.147:50052: connect: connection refused"
Nov 03 01:14:26.333735 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:14:26.333707    6000 logging.go:55] [core] [Channel #19963 SubChannel #19965]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.148:50052", ServerName: "10.134.0.148:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.148:50052: connect: connection refused"
Nov 03 01:14:26.524158 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.524129    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 01:14:26.550299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.550276    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 01:14:26.615015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.613747    6000 generic.go:356] "Generic (PLEG): container finished" podID="b45eccb1-a80a-4972-9293-b569f843f19f" containerID="5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f" exitCode=0
Nov 03 01:14:26.615015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.613984    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" event={"ID":"b45eccb1-a80a-4972-9293-b569f843f19f","Type":"ContainerDied","Data":"5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f"}
Nov 03 01:14:26.615015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.614040    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" event={"ID":"b45eccb1-a80a-4972-9293-b569f843f19f","Type":"ContainerDied","Data":"d8601187f96b3a81ddddf8aaa33866838a9dc7b03b36bffe3c6acf596ff23ca7"}
Nov 03 01:14:26.615015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.614081    6000 scope.go:122] "RemoveContainer" containerID="5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f"
Nov 03 01:14:26.615015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.614276    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"
Nov 03 01:14:26.619961 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.619942    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"
Nov 03 01:14:26.620341 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.619968    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" event={"ID":"bfe2e57d-7713-4ca2-afb5-14c11fd739f9","Type":"ContainerDied","Data":"7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb"}
Nov 03 01:14:26.620497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.619822    6000 generic.go:356] "Generic (PLEG): container finished" podID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerID="7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb" exitCode=0
Nov 03 01:14:26.620651 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.620633    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" event={"ID":"bfe2e57d-7713-4ca2-afb5-14c11fd739f9","Type":"ContainerDied","Data":"37cccc0e69fa643f10306d80a2a5f7f5519579a58cd50eb3ff4a261f44c8740f"}
Nov 03 01:14:26.630033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.630012    6000 scope.go:122] "RemoveContainer" containerID="5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f"
Nov 03 01:14:26.630410 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:26.630379    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f\": container with ID starting with 5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f not found: ID does not exist" containerID="5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f"
Nov 03 01:14:26.630503 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.630420    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f"} err="failed to get container status \"5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f\": rpc error: code = NotFound desc = could not find container \"5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f\": container with ID starting with 5609fb8ae2b5b5dac901a3f1810c99faa1e942ed97c06f39c588440e2d415d6f not found: ID does not exist"
Nov 03 01:14:26.630503 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.630442    6000 scope.go:122] "RemoveContainer" containerID="7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb"
Nov 03 01:14:26.640350 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.640323    6000 scope.go:122] "RemoveContainer" containerID="7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb"
Nov 03 01:14:26.640647 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:26.640617    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb\": container with ID starting with 7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb not found: ID does not exist" containerID="7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb"
Nov 03 01:14:26.640751 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.640658    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb"} err="failed to get container status \"7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb\": rpc error: code = NotFound desc = could not find container \"7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb\": container with ID starting with 7696b81f52aec62bc57f58344b9c4557382587ed21bdbdc1028cfd821326facb not found: ID does not exist"
Nov 03 01:14:26.658091 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658069    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b45eccb1-a80a-4972-9293-b569f843f19f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45eccb1-a80a-4972-9293-b569f843f19f-plugins-config-volume\") pod \"b45eccb1-a80a-4972-9293-b569f843f19f\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") "
Nov 03 01:14:26.658190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658131    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-plugins-config-volume\") pod \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") "
Nov 03 01:14:26.658190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658169    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-kube-api-access-m8v2k\" (UniqueName: \"kubernetes.io/projected/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-kube-api-access-m8v2k\") pod \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\" (UID: \"bfe2e57d-7713-4ca2-afb5-14c11fd739f9\") "
Nov 03 01:14:26.658333 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658200    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b45eccb1-a80a-4972-9293-b569f843f19f-kube-api-access-bg72l\" (UniqueName: \"kubernetes.io/projected/b45eccb1-a80a-4972-9293-b569f843f19f-kube-api-access-bg72l\") pod \"b45eccb1-a80a-4972-9293-b569f843f19f\" (UID: \"b45eccb1-a80a-4972-9293-b569f843f19f\") "
Nov 03 01:14:26.658520 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658492    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b45eccb1-a80a-4972-9293-b569f843f19f-plugins-config-volume" pod "b45eccb1-a80a-4972-9293-b569f843f19f" (UID: "b45eccb1-a80a-4972-9293-b569f843f19f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:26.658613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658579    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-plugins-config-volume" pod "bfe2e57d-7713-4ca2-afb5-14c11fd739f9" (UID: "bfe2e57d-7713-4ca2-afb5-14c11fd739f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:26.658935 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658908    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b45eccb1-a80a-4972-9293-b569f843f19f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:26.659034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.658940    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:26.660844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.660820    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b45eccb1-a80a-4972-9293-b569f843f19f-kube-api-access-bg72l" pod "b45eccb1-a80a-4972-9293-b569f843f19f" (UID: "b45eccb1-a80a-4972-9293-b569f843f19f"). InnerVolumeSpecName "kube-api-access-bg72l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:26.660942 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.660847    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-kube-api-access-m8v2k" pod "bfe2e57d-7713-4ca2-afb5-14c11fd739f9" (UID: "bfe2e57d-7713-4ca2-afb5-14c11fd739f9"). InnerVolumeSpecName "kube-api-access-m8v2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:26.760200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.760124    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8v2k\" (UniqueName: \"kubernetes.io/projected/bfe2e57d-7713-4ca2-afb5-14c11fd739f9-kube-api-access-m8v2k\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:26.760200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.760152    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bg72l\" (UniqueName: \"kubernetes.io/projected/b45eccb1-a80a-4972-9293-b569f843f19f-kube-api-access-bg72l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:26.956101 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:26.956066    6000 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb45eccb1_a80a_4972_9293_b569f843f19f.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe2e57d_7713_4ca2_afb5_14c11fd739f9.slice/crio-37cccc0e69fa643f10306d80a2a5f7f5519579a58cd50eb3ff4a261f44c8740f\": RecentStats: unable to find data in memory cache]"
Nov 03 01:14:26.960893 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.960862    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"]
Nov 03 01:14:26.970792 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.970765    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs"]
Nov 03 01:14:26.993660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.993634    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"]
Nov 03 01:14:26.996800 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:26.996770    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s"]
Nov 03 01:14:27.333916 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:27.333867    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-065de0-656c4dc6b5-m7n2s" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.148:50052\" within 1s: context deadline exceeded"
Nov 03 01:14:27.333916 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:27.333904    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-065de0-7b998bf94c-w5scs" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.147:50052\" within 1s: context deadline exceeded"
Nov 03 01:14:28.568454 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:28.568420    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" path="/var/lib/kubelet/pods/b45eccb1-a80a-4972-9293-b569f843f19f/volumes"
Nov 03 01:14:28.568902 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:28.568864    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" path="/var/lib/kubelet/pods/bfe2e57d-7713-4ca2-afb5-14c11fd739f9/volumes"
Nov 03 01:14:29.645587 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.645550    6000 generic.go:356] "Generic (PLEG): container finished" podID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerID="8e20ed78bfd07682ebe0fb77fb068e5b2b007261dc6a83651d51c38475add2ca" exitCode=0
Nov 03 01:14:29.646010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.645622    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" event={"ID":"0155e021-c45b-4fc5-a94e-a2df797f965f","Type":"ContainerDied","Data":"8e20ed78bfd07682ebe0fb77fb068e5b2b007261dc6a83651d51c38475add2ca"}
Nov 03 01:14:29.646010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.645674    6000 scope.go:122] "RemoveContainer" containerID="352aba70745a9a63c4bd6b014a199fad79ec4cb612410abf2721e06f7ad85fd5"
Nov 03 01:14:29.887219 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.887189    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 01:14:29.996318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996231    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-certs\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996486 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996316    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-podinfo\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996486 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996369    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0155e021-c45b-4fc5-a94e-a2df797f965f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0155e021-c45b-4fc5-a94e-a2df797f965f-istiod-ca-cert\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996486 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996406    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-envoy\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996486 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996460    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-token\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-token\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996692 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996565    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-certs" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:29.996692 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996580    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-data\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996692 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996647    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-credential-socket\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996866 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996798    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-kube-api-access-5fg5j\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-kube-api-access-5fg5j\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.996866 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996825    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-data" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:29.996866 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996833    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-socket\") pod \"0155e021-c45b-4fc5-a94e-a2df797f965f\" (UID: \"0155e021-c45b-4fc5-a94e-a2df797f965f\") "
Nov 03 01:14:29.997053 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996962    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-credential-socket" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:29.997053 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.996988    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0155e021-c45b-4fc5-a94e-a2df797f965f-istiod-ca-cert" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:14:29.997220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.997191    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-socket" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:29.997447 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.997401    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:29.997571 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.997431    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:29.997660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.997527    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:29.997660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.997623    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:29.997660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.997637    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0155e021-c45b-4fc5-a94e-a2df797f965f-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:29.998901 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.998879    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-podinfo" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:14:29.999162 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.999143    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-envoy" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:14:29.999285 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.999231    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-token" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:29.999369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:29.999352    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-kube-api-access-5fg5j" pod "0155e021-c45b-4fc5-a94e-a2df797f965f" (UID: "0155e021-c45b-4fc5-a94e-a2df797f965f"). InnerVolumeSpecName "kube-api-access-5fg5j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:30.098722 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.098685    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:30.098722 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.098715    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:30.098722 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.098724    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:30.098982 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.098733    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5fg5j\" (UniqueName: \"kubernetes.io/projected/0155e021-c45b-4fc5-a94e-a2df797f965f-kube-api-access-5fg5j\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:30.653744 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.653695    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v" event={"ID":"0155e021-c45b-4fc5-a94e-a2df797f965f","Type":"ContainerDied","Data":"854a0d8a32096a5a3cfca1de414885f53c8eddc15353deaf0f827984349a0538"}
Nov 03 01:14:30.654177 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.653782    6000 scope.go:122] "RemoveContainer" containerID="8e20ed78bfd07682ebe0fb77fb068e5b2b007261dc6a83651d51c38475add2ca"
Nov 03 01:14:30.654177 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.653714    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"
Nov 03 01:14:30.718762 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.718732    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"]
Nov 03 01:14:30.729857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:30.729830    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-065de0-openshift-default-64f779b7-hkv7v"]
Nov 03 01:14:32.568284 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:32.568235    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" path="/var/lib/kubelet/pods/0155e021-c45b-4fc5-a94e-a2df797f965f/volumes"
Nov 03 01:14:39.986976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:39.986936    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:14:44.580761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.580730    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-6f4f1c"]
Nov 03 01:14:44.581297 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581282    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy"
Nov 03 01:14:44.581348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581297    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy"
Nov 03 01:14:44.581348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581323    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy"
Nov 03 01:14:44.581348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581328    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy"
Nov 03 01:14:44.581348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581340    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy"
Nov 03 01:14:44.581348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581346    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581352    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581358    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581366    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c7c6efed-66cc-43c5-b23c-2b09f33508a5" containerName="maas-api"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581371    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c6efed-66cc-43c5-b23c-2b09f33508a5" containerName="maas-api"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581383    6000 cpu_manager.go:414] "containerMap: removing container" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581387    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581403    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerName="payload-processing"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581408    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerName="payload-processing"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581418    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581423    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581436    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" containerName="payload-pre-processing"
Nov 03 01:14:44.581513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581440    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" containerName="payload-pre-processing"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581522    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfe2e57d-7713-4ca2-afb5-14c11fd739f9" containerName="payload-processing"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581531    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a1e7b0f-2b36-4171-8233-395051e2fb8d" containerName="istio-proxy"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581539    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581545    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b45eccb1-a80a-4972-9293-b569f843f19f" containerName="payload-pre-processing"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581550    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d63df94-9b02-4c1b-aae4-1e52e57b5d9e" containerName="istio-proxy"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581556    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d54cd764-315b-4333-960e-1b442ed0ab93" containerName="istio-proxy"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581564    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581569    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b5e4b26-c21e-4fde-b787-33bb4685879b" containerName="istio-proxy"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581576    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c6efed-66cc-43c5-b23c-2b09f33508a5" containerName="maas-api"
Nov 03 01:14:44.581931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.581582    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="82f4fb1a-2c68-49ce-9cc6-8c18e4949548" containerName="istio-proxy"
Nov 03 01:14:44.584984 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.584968    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:44.601307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.601279    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-6f4f1c"]
Nov 03 01:14:44.610492 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.610466    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-b5fmb\""
Nov 03 01:14:44.668774 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.668699    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t79g\" (UniqueName: \"kubernetes.io/projected/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8-kube-api-access-6t79g\") pod \"test-curl-7609-6f4f1c\" (UID: \"c5eb4232-02cf-4274-b1ab-2aa58ba49fc8\") " pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:44.769993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.769961    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t79g\" (UniqueName: \"kubernetes.io/projected/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8-kube-api-access-6t79g\") pod \"test-curl-7609-6f4f1c\" (UID: \"c5eb4232-02cf-4274-b1ab-2aa58ba49fc8\") " pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:44.788398 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.788373    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t79g\" (UniqueName: \"kubernetes.io/projected/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8-kube-api-access-6t79g\") pod \"test-curl-7609-6f4f1c\" (UID: \"c5eb4232-02cf-4274-b1ab-2aa58ba49fc8\") " pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:44.897725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:44.897699    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:45.189190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:45.189153    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-6f4f1c"]
Nov 03 01:14:45.190719 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:14:45.190691    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5eb4232_02cf_4274_b1ab_2aa58ba49fc8.slice/crio-0877bf9bf1fcf89544b8eefd19b8f1134c9191a23a0713fe740d93c457ebcee8 WatchSource:0}: Error finding container 0877bf9bf1fcf89544b8eefd19b8f1134c9191a23a0713fe740d93c457ebcee8: Status 404 returned error can't find the container with id 0877bf9bf1fcf89544b8eefd19b8f1134c9191a23a0713fe740d93c457ebcee8
Nov 03 01:14:45.750342 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:45.750300    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-6f4f1c" event={"ID":"c5eb4232-02cf-4274-b1ab-2aa58ba49fc8","Type":"ContainerStarted","Data":"37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb"}
Nov 03 01:14:45.750773 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:45.750356    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-6f4f1c" event={"ID":"c5eb4232-02cf-4274-b1ab-2aa58ba49fc8","Type":"ContainerStarted","Data":"0877bf9bf1fcf89544b8eefd19b8f1134c9191a23a0713fe740d93c457ebcee8"}
Nov 03 01:14:45.785750 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:45.785703    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-6f4f1c" podStartSLOduration=1.785690288 podStartE2EDuration="1.785690288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:45.783849217 +0000 UTC m=+4123.339201239" watchObservedRunningTime="2026-11-03 01:14:45.785690288 +0000 UTC m=+4123.341042307"
Nov 03 01:14:46.629562 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.629524    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-6f4f1c"]
Nov 03 01:14:46.635858 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.635820    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-6f4f1c"]
Nov 03 01:14:46.783922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.783885    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-981f08"]
Nov 03 01:14:46.784474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.784459    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy"
Nov 03 01:14:46.784530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.784475    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0155e021-c45b-4fc5-a94e-a2df797f965f" containerName="istio-proxy"
Nov 03 01:14:46.784530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.784510    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" containerName="test-curl-7609-6f4f1c"
Nov 03 01:14:46.784530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.784516    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" containerName="test-curl-7609-6f4f1c"
Nov 03 01:14:46.784629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.784623    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" containerName="test-curl-7609-6f4f1c"
Nov 03 01:14:46.787864 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.787847    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:46.805766 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.805735    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-981f08"]
Nov 03 01:14:46.894399 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.894313    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2wl\" (UniqueName: \"kubernetes.io/projected/24f41056-f5f3-47f7-a917-61b13d3925b7-kube-api-access-8j2wl\") pod \"test-curl-7609-981f08\" (UID: \"24f41056-f5f3-47f7-a917-61b13d3925b7\") " pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:46.995120 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:46.995079    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2wl\" (UniqueName: \"kubernetes.io/projected/24f41056-f5f3-47f7-a917-61b13d3925b7-kube-api-access-8j2wl\") pod \"test-curl-7609-981f08\" (UID: \"24f41056-f5f3-47f7-a917-61b13d3925b7\") " pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:47.013453 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:47.013420    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2wl\" (UniqueName: \"kubernetes.io/projected/24f41056-f5f3-47f7-a917-61b13d3925b7-kube-api-access-8j2wl\") pod \"test-curl-7609-981f08\" (UID: \"24f41056-f5f3-47f7-a917-61b13d3925b7\") " pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:47.100219 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:47.100184    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:47.393379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:47.393343    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-981f08"]
Nov 03 01:14:47.394628 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:14:47.394596    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24f41056_f5f3_47f7_a917_61b13d3925b7.slice/crio-2479398c177af0d7bece0fee2abc7076de511adabecc788d407053f84f26b2cb WatchSource:0}: Error finding container 2479398c177af0d7bece0fee2abc7076de511adabecc788d407053f84f26b2cb: Status 404 returned error can't find the container with id 2479398c177af0d7bece0fee2abc7076de511adabecc788d407053f84f26b2cb
Nov 03 01:14:47.764198 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:47.764094    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-981f08" event={"ID":"24f41056-f5f3-47f7-a917-61b13d3925b7","Type":"ContainerStarted","Data":"b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b"}
Nov 03 01:14:47.764198 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:47.764144    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-981f08" event={"ID":"24f41056-f5f3-47f7-a917-61b13d3925b7","Type":"ContainerStarted","Data":"2479398c177af0d7bece0fee2abc7076de511adabecc788d407053f84f26b2cb"}
Nov 03 01:14:47.764493 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:47.764364    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-6f4f1c" podUID="c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" containerName="test-curl-7609-6f4f1c" containerID="cri-o://37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb" gracePeriod=2
Nov 03 01:14:47.799596 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:47.799550    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7609-981f08" podStartSLOduration=1.7995372029999999 podStartE2EDuration="1.799537203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:47.799022067 +0000 UTC m=+4125.354374081" watchObservedRunningTime="2026-11-03 01:14:47.799537203 +0000 UTC m=+4125.354889227"
Nov 03 01:14:47.811643 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:47.811617    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7609-6f4f1c\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:48.676312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:48.676280    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-981f08"]
Nov 03 01:14:48.680860 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:48.680817    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-981f08"]
Nov 03 01:14:49.069173 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.069092    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"]
Nov 03 01:14:49.069871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.069846    6000 cpu_manager.go:414] "containerMap: removing container" podUID="24f41056-f5f3-47f7-a917-61b13d3925b7" containerName="test-curl-7609-981f08"
Nov 03 01:14:49.069871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.069871    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24f41056-f5f3-47f7-a917-61b13d3925b7" containerName="test-curl-7609-981f08"
Nov 03 01:14:49.070065 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.070026    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="24f41056-f5f3-47f7-a917-61b13d3925b7" containerName="test-curl-7609-981f08"
Nov 03 01:14:49.073650 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.073629    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.087952 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.087931    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-20109d-openshift-default-dockercfg-ls98l\""
Nov 03 01:14:49.102575 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.102547    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"]
Nov 03 01:14:49.120348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120310    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-podinfo\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120364    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120487    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istiod-ca-cert\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120535    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-envoy\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120667 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120600    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr6sm\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-kube-api-access-lr6sm\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120691    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-credential-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120799 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120778    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-token\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120878 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120863    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-certs\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.120938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.120897    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-data\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222127 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222088    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-podinfo\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222133    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222197    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istiod-ca-cert\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222239    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-envoy\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222303    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr6sm\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-kube-api-access-lr6sm\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222345    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-credential-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222390    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-token\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222454    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-certs\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222484    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-data\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.222818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.222720    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.223119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.223093    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-credential-socket\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.223119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.223113    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-data\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.223292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.223108    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istiod-ca-cert\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.223734 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.223339    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-certs\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.224933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.224913    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-podinfo\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.225155 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.225139    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-envoy\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.243467 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.243441    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-token\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.243907 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.243881    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr6sm\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-kube-api-access-lr6sm\") pod \"e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") " pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.388463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.388426    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:49.699292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.699231    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"]
Nov 03 01:14:49.699896 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:14:49.699866    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ab309f_7e1a_4789_b54e_f609b6c94ad6.slice/crio-740332da218db14f67a77bab3ff9e2d8e6082c13bcb391b4914c8aac0cf813f4 WatchSource:0}: Error finding container 740332da218db14f67a77bab3ff9e2d8e6082c13bcb391b4914c8aac0cf813f4: Status 404 returned error can't find the container with id 740332da218db14f67a77bab3ff9e2d8e6082c13bcb391b4914c8aac0cf813f4
Nov 03 01:14:49.702629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.702594    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:14:49.702726 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.702659    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:14:49.702726 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.702687    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:14:49.778659 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.778631    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" event={"ID":"09ab309f-7e1a-4789-b54e-f609b6c94ad6","Type":"ContainerStarted","Data":"740332da218db14f67a77bab3ff9e2d8e6082c13bcb391b4914c8aac0cf813f4"}
Nov 03 01:14:49.778885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.778836    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7609-981f08" podUID="24f41056-f5f3-47f7-a917-61b13d3925b7" containerName="test-curl-7609-981f08" containerID="cri-o://b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b" gracePeriod=2
Nov 03 01:14:49.791467 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:49.791441    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7609-981f08\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="24f41056-f5f3-47f7-a917-61b13d3925b7" pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:49.917419 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:49.916559    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:14:49.955418 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:49.955390    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:49.968387 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:49.968359    6000 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7609-981f08\" is forbidden: User \"system:node:ip-10-0-136-97.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-136-97.ec2.internal' and this object" podUID="24f41056-f5f3-47f7-a917-61b13d3925b7" pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:50.033196 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.033158    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8-kube-api-access-6t79g\" (UniqueName: \"kubernetes.io/projected/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8-kube-api-access-6t79g\") pod \"c5eb4232-02cf-4274-b1ab-2aa58ba49fc8\" (UID: \"c5eb4232-02cf-4274-b1ab-2aa58ba49fc8\") "
Nov 03 01:14:50.035982 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.035948    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8-kube-api-access-6t79g" pod "c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" (UID: "c5eb4232-02cf-4274-b1ab-2aa58ba49fc8"). InnerVolumeSpecName "kube-api-access-6t79g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:50.134646 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.134610    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t79g\" (UniqueName: \"kubernetes.io/projected/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8-kube-api-access-6t79g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:50.568513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.568431    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" path="/var/lib/kubelet/pods/c5eb4232-02cf-4274-b1ab-2aa58ba49fc8/volumes"
Nov 03 01:14:50.785976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.785938    6000 generic.go:356] "Generic (PLEG): container finished" podID="c5eb4232-02cf-4274-b1ab-2aa58ba49fc8" containerID="37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb" exitCode=137
Nov 03 01:14:50.786163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.785985    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-6f4f1c"
Nov 03 01:14:50.786163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.786036    6000 scope.go:122] "RemoveContainer" containerID="37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb"
Nov 03 01:14:50.814571 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.814534    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" event={"ID":"09ab309f-7e1a-4789-b54e-f609b6c94ad6","Type":"ContainerStarted","Data":"7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116"}
Nov 03 01:14:50.821663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.821639    6000 scope.go:122] "RemoveContainer" containerID="37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb"
Nov 03 01:14:50.821926 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:50.821906    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb\": container with ID starting with 37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb not found: ID does not exist" containerID="37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb"
Nov 03 01:14:50.821975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.821935    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb"} err="failed to get container status \"37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb\": rpc error: code = NotFound desc = could not find container \"37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb\": container with ID starting with 37b24bd13edd43aa359c64766a64f7cd126e6334d1a0571405de27b06d7b1afb not found: ID does not exist"
Nov 03 01:14:50.876353 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:50.876294    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" podStartSLOduration=1.876276629 podStartE2EDuration="1.876276629s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:14:50.864722084 +0000 UTC m=+4128.420074120" watchObservedRunningTime="2026-11-03 01:14:50.876276629 +0000 UTC m=+4128.431628648"
Nov 03 01:14:51.394627 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:51.394592    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:51.411482 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:51.411451    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Nov 03 01:14:51.411787 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:51.411751    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Nov 03 01:14:52.028156 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.028126    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:52.161701 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.161668    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24f41056-f5f3-47f7-a917-61b13d3925b7-kube-api-access-8j2wl\" (UniqueName: \"kubernetes.io/projected/24f41056-f5f3-47f7-a917-61b13d3925b7-kube-api-access-8j2wl\") pod \"24f41056-f5f3-47f7-a917-61b13d3925b7\" (UID: \"24f41056-f5f3-47f7-a917-61b13d3925b7\") "
Nov 03 01:14:52.163995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.163966    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24f41056-f5f3-47f7-a917-61b13d3925b7-kube-api-access-8j2wl" pod "24f41056-f5f3-47f7-a917-61b13d3925b7" (UID: "24f41056-f5f3-47f7-a917-61b13d3925b7"). InnerVolumeSpecName "kube-api-access-8j2wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:14:52.263294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.263238    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8j2wl\" (UniqueName: \"kubernetes.io/projected/24f41056-f5f3-47f7-a917-61b13d3925b7-kube-api-access-8j2wl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:14:52.400283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.393657    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Nov 03 01:14:52.400283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.393724    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Nov 03 01:14:52.567890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.567813    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24f41056-f5f3-47f7-a917-61b13d3925b7" path="/var/lib/kubelet/pods/24f41056-f5f3-47f7-a917-61b13d3925b7/volumes"
Nov 03 01:14:52.833182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.833104    6000 generic.go:356] "Generic (PLEG): container finished" podID="24f41056-f5f3-47f7-a917-61b13d3925b7" containerID="b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b" exitCode=137
Nov 03 01:14:52.833182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.833162    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-981f08"
Nov 03 01:14:52.833428 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.833207    6000 scope.go:122] "RemoveContainer" containerID="b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b"
Nov 03 01:14:52.842750 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.842730    6000 scope.go:122] "RemoveContainer" containerID="b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b"
Nov 03 01:14:52.843049 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:14:52.843030    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b\": container with ID starting with b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b not found: ID does not exist" containerID="b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b"
Nov 03 01:14:52.843121 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:52.843056    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b"} err="failed to get container status \"b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b\": rpc error: code = NotFound desc = could not find container \"b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b\": container with ID starting with b1e2450bfe39f9bbf441b7f430d5bbd320403480cf534efd6ac3b692b97c2c9b not found: ID does not exist"
Nov 03 01:14:53.403727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:53.403686    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Nov 03 01:14:53.404402 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:53.404371    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Nov 03 01:14:54.394699 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:54.394657    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Nov 03 01:14:54.394999 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:54.394971    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Nov 03 01:14:55.412429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:55.412385    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused" start-of-body=
Nov 03 01:14:55.412968 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:55.412455    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.40:15021/healthz/ready\": dial tcp 10.134.1.40:15021: connect: connection refused"
Nov 03 01:14:56.393526 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:56.393495    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:56.393975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:56.393956    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:56.394773 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:56.394756    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:14:59.987117 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:14:59.987075    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"]
Nov 03 01:15:00.157796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.157759    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"]
Nov 03 01:15:00.163581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.163554    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.192919 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.192893    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gxqh2\""
Nov 03 01:15:00.193575 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.193546    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"]
Nov 03 01:15:00.199435 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.199411    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"]
Nov 03 01:15:00.199558 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.199536    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.213299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.213272    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"]
Nov 03 01:15:00.217615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.217589    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.224816 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.224728    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"]
Nov 03 01:15:00.231165 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.231142    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"]
Nov 03 01:15:00.236473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.236449    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.237960 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.237890    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"]
Nov 03 01:15:00.252958 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.252931    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"]
Nov 03 01:15:00.277758 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.277723    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5lv9\" (UniqueName: \"kubernetes.io/projected/508ab7fa-8fe3-4390-8a38-c40d94180c3e-kube-api-access-f5lv9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.277906 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.277804    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ab7fa-8fe3-4390-8a38-c40d94180c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.379541 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379504    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991686e6-13bc-41c3-a577-baf8686c1109-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.379720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379559    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5lv9\" (UniqueName: \"kubernetes.io/projected/508ab7fa-8fe3-4390-8a38-c40d94180c3e-kube-api-access-f5lv9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.379720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379641    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ab7fa-8fe3-4390-8a38-c40d94180c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.379720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379717    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nm2f\" (UniqueName: \"kubernetes.io/projected/991686e6-13bc-41c3-a577-baf8686c1109-kube-api-access-9nm2f\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.379883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379741    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7551f80e-1fa7-4afd-acf5-f0364d517ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.379883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379763    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtl8k\" (UniqueName: \"kubernetes.io/projected/7551f80e-1fa7-4afd-acf5-f0364d517ac6-kube-api-access-jtl8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.379883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379796    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba47343d-7e39-4e7b-855a-04edb08f8ba1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.379994 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.379945    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btn9p\" (UniqueName: \"kubernetes.io/projected/ba47343d-7e39-4e7b-855a-04edb08f8ba1-kube-api-access-btn9p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.380342 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.380326    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ab7fa-8fe3-4390-8a38-c40d94180c3e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.398706 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.398675    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5lv9\" (UniqueName: \"kubernetes.io/projected/508ab7fa-8fe3-4390-8a38-c40d94180c3e-kube-api-access-f5lv9\") pod \"maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.477435 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.477403    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:00.480972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.480662    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nm2f\" (UniqueName: \"kubernetes.io/projected/991686e6-13bc-41c3-a577-baf8686c1109-kube-api-access-9nm2f\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.480972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.480698    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7551f80e-1fa7-4afd-acf5-f0364d517ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.480972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.480723    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtl8k\" (UniqueName: \"kubernetes.io/projected/7551f80e-1fa7-4afd-acf5-f0364d517ac6-kube-api-access-jtl8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.480972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.480853    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba47343d-7e39-4e7b-855a-04edb08f8ba1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.480972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.480937    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btn9p\" (UniqueName: \"kubernetes.io/projected/ba47343d-7e39-4e7b-855a-04edb08f8ba1-kube-api-access-btn9p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.481363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.480987    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991686e6-13bc-41c3-a577-baf8686c1109-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.481512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.481474    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7551f80e-1fa7-4afd-acf5-f0364d517ac6-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.481632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.481588    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba47343d-7e39-4e7b-855a-04edb08f8ba1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.481722 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.481701    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991686e6-13bc-41c3-a577-baf8686c1109-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.501995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.501965    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtl8k\" (UniqueName: \"kubernetes.io/projected/7551f80e-1fa7-4afd-acf5-f0364d517ac6-kube-api-access-jtl8k\") pod \"maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.502472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.502451    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btn9p\" (UniqueName: \"kubernetes.io/projected/ba47343d-7e39-4e7b-855a-04edb08f8ba1-kube-api-access-btn9p\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.502579 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.502520    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nm2f\" (UniqueName: \"kubernetes.io/projected/991686e6-13bc-41c3-a577-baf8686c1109-kube-api-access-9nm2f\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.517035 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.517008    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:00.539568 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.538481    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:00.558075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.557834    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:00.778127 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.778097    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"]
Nov 03 01:15:00.778581 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:15:00.778560    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod508ab7fa_8fe3_4390_8a38_c40d94180c3e.slice/crio-ff1b5f805a87d423eb25f8859fb58e46c2fbc5292e126d55675bc46d22cf17f0 WatchSource:0}: Error finding container ff1b5f805a87d423eb25f8859fb58e46c2fbc5292e126d55675bc46d22cf17f0: Status 404 returned error can't find the container with id ff1b5f805a87d423eb25f8859fb58e46c2fbc5292e126d55675bc46d22cf17f0
Nov 03 01:15:00.847518 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.846887    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"]
Nov 03 01:15:00.847518 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:15:00.847390    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba47343d_7e39_4e7b_855a_04edb08f8ba1.slice/crio-ce5ed95ca5d3ab57f88cb363babbb00cc93560726a11eaa0d0d08e5d2bc6e7d0 WatchSource:0}: Error finding container ce5ed95ca5d3ab57f88cb363babbb00cc93560726a11eaa0d0d08e5d2bc6e7d0: Status 404 returned error can't find the container with id ce5ed95ca5d3ab57f88cb363babbb00cc93560726a11eaa0d0d08e5d2bc6e7d0
Nov 03 01:15:00.869503 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.868972    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"]
Nov 03 01:15:00.888220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.888185    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" event={"ID":"508ab7fa-8fe3-4390-8a38-c40d94180c3e","Type":"ContainerStarted","Data":"ff1b5f805a87d423eb25f8859fb58e46c2fbc5292e126d55675bc46d22cf17f0"}
Nov 03 01:15:00.891947 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.891917    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"]
Nov 03 01:15:00.892980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.892956    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" event={"ID":"7551f80e-1fa7-4afd-acf5-f0364d517ac6","Type":"ContainerStarted","Data":"2716243c83ea20bad34f0187cddc06a1e29139426caa567444cf8d66bb1b497b"}
Nov 03 01:15:00.894418 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:00.894395    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" event={"ID":"ba47343d-7e39-4e7b-855a-04edb08f8ba1","Type":"ContainerStarted","Data":"ce5ed95ca5d3ab57f88cb363babbb00cc93560726a11eaa0d0d08e5d2bc6e7d0"}
Nov 03 01:15:00.896577 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:15:00.896557    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod991686e6_13bc_41c3_a577_baf8686c1109.slice/crio-e18c02f794b5d9df7bd0c93deb84ab7e032ff2768e2480bd023351d898884d52 WatchSource:0}: Error finding container e18c02f794b5d9df7bd0c93deb84ab7e032ff2768e2480bd023351d898884d52: Status 404 returned error can't find the container with id e18c02f794b5d9df7bd0c93deb84ab7e032ff2768e2480bd023351d898884d52
Nov 03 01:15:01.904080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.904048    6000 generic.go:356] "Generic (PLEG): container finished" podID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerID="ec3afc589b9cd969e9ff08319df1b57c82bf627959014727456e9a7dca924025" exitCode=60
Nov 03 01:15:01.904553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.904137    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" event={"ID":"508ab7fa-8fe3-4390-8a38-c40d94180c3e","Type":"ContainerDied","Data":"ec3afc589b9cd969e9ff08319df1b57c82bf627959014727456e9a7dca924025"}
Nov 03 01:15:01.904553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.904479    6000 scope.go:122] "RemoveContainer" containerID="ec3afc589b9cd969e9ff08319df1b57c82bf627959014727456e9a7dca924025"
Nov 03 01:15:01.910775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.910751    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" event={"ID":"7551f80e-1fa7-4afd-acf5-f0364d517ac6","Type":"ContainerStarted","Data":"834e095b6d45efef49eb0de2b8b42f17f96a624b0e8369bcc4b46e1a44e2152d"}
Nov 03 01:15:01.913204 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.913178    6000 generic.go:356] "Generic (PLEG): container finished" podID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerID="28e45991cfb9b2c8623cf3fc135dd980e7f415722d89ba9b9ec32ded431a4f95" exitCode=60
Nov 03 01:15:01.913346 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.913236    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" event={"ID":"ba47343d-7e39-4e7b-855a-04edb08f8ba1","Type":"ContainerDied","Data":"28e45991cfb9b2c8623cf3fc135dd980e7f415722d89ba9b9ec32ded431a4f95"}
Nov 03 01:15:01.913500 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.913484    6000 scope.go:122] "RemoveContainer" containerID="28e45991cfb9b2c8623cf3fc135dd980e7f415722d89ba9b9ec32ded431a4f95"
Nov 03 01:15:01.915030 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.915009    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" event={"ID":"991686e6-13bc-41c3-a577-baf8686c1109","Type":"ContainerStarted","Data":"1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638"}
Nov 03 01:15:01.915147 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.915044    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" event={"ID":"991686e6-13bc-41c3-a577-baf8686c1109","Type":"ContainerStarted","Data":"e18c02f794b5d9df7bd0c93deb84ab7e032ff2768e2480bd023351d898884d52"}
Nov 03 01:15:01.915231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:01.915214    6000 scope.go:122] "RemoveContainer" containerID="1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638"
Nov 03 01:15:02.020062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.020015    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" podStartSLOduration=2.020001084 podStartE2EDuration="2.020001084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:02.017358017 +0000 UTC m=+4139.572710041" watchObservedRunningTime="2026-11-03 01:15:02.020001084 +0000 UTC m=+4139.575353101"
Nov 03 01:15:02.923272 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.923219    6000 generic.go:356] "Generic (PLEG): container finished" podID="991686e6-13bc-41c3-a577-baf8686c1109" containerID="1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638" exitCode=60
Nov 03 01:15:02.923272 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.923271    6000 generic.go:356] "Generic (PLEG): container finished" podID="991686e6-13bc-41c3-a577-baf8686c1109" containerID="67c224f997f1faf3596a531cd37555828c8b2f8e6a49918a12787bd812b62094" exitCode=60
Nov 03 01:15:02.923745 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.923342    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" event={"ID":"991686e6-13bc-41c3-a577-baf8686c1109","Type":"ContainerDied","Data":"1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638"}
Nov 03 01:15:02.923745 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.923405    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" event={"ID":"991686e6-13bc-41c3-a577-baf8686c1109","Type":"ContainerDied","Data":"67c224f997f1faf3596a531cd37555828c8b2f8e6a49918a12787bd812b62094"}
Nov 03 01:15:02.923745 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.923426    6000 scope.go:122] "RemoveContainer" containerID="1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638"
Nov 03 01:15:02.923745 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.923702    6000 scope.go:122] "RemoveContainer" containerID="67c224f997f1faf3596a531cd37555828c8b2f8e6a49918a12787bd812b62094"
Nov 03 01:15:02.923989 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:02.923968    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7_odh-ai-gateway-infra(991686e6-13bc-41c3-a577-baf8686c1109)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" podUID="991686e6-13bc-41c3-a577-baf8686c1109"
Nov 03 01:15:02.925406 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.925386    6000 generic.go:356] "Generic (PLEG): container finished" podID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerID="3557b094adc95afbbc4b27133c2be8c961a82c2dc3467bef1b61c7dbe2601bad" exitCode=60
Nov 03 01:15:02.925508 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.925448    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" event={"ID":"508ab7fa-8fe3-4390-8a38-c40d94180c3e","Type":"ContainerDied","Data":"3557b094adc95afbbc4b27133c2be8c961a82c2dc3467bef1b61c7dbe2601bad"}
Nov 03 01:15:02.925709 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.925692    6000 scope.go:122] "RemoveContainer" containerID="3557b094adc95afbbc4b27133c2be8c961a82c2dc3467bef1b61c7dbe2601bad"
Nov 03 01:15:02.925986 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:02.925961    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw_odh-ai-gateway-infra(508ab7fa-8fe3-4390-8a38-c40d94180c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e"
Nov 03 01:15:02.927071 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.927051    6000 generic.go:356] "Generic (PLEG): container finished" podID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerID="834e095b6d45efef49eb0de2b8b42f17f96a624b0e8369bcc4b46e1a44e2152d" exitCode=60
Nov 03 01:15:02.927179 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.927137    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" event={"ID":"7551f80e-1fa7-4afd-acf5-f0364d517ac6","Type":"ContainerDied","Data":"834e095b6d45efef49eb0de2b8b42f17f96a624b0e8369bcc4b46e1a44e2152d"}
Nov 03 01:15:02.927376 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.927355    6000 scope.go:122] "RemoveContainer" containerID="834e095b6d45efef49eb0de2b8b42f17f96a624b0e8369bcc4b46e1a44e2152d"
Nov 03 01:15:02.942975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.942948    6000 generic.go:356] "Generic (PLEG): container finished" podID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerID="9825fc5c394484b4243bb89545dbdaaf9fba250bd2a8a3900ec33f1f69cd90e8" exitCode=60
Nov 03 01:15:02.943183 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.943029    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" event={"ID":"ba47343d-7e39-4e7b-855a-04edb08f8ba1","Type":"ContainerDied","Data":"9825fc5c394484b4243bb89545dbdaaf9fba250bd2a8a3900ec33f1f69cd90e8"}
Nov 03 01:15:02.943399 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.943348    6000 scope.go:122] "RemoveContainer" containerID="9825fc5c394484b4243bb89545dbdaaf9fba250bd2a8a3900ec33f1f69cd90e8"
Nov 03 01:15:02.943986 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:02.943958    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx_odh-ai-gateway-infra(ba47343d-7e39-4e7b-855a-04edb08f8ba1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1"
Nov 03 01:15:02.998518 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.998496    6000 scope.go:122] "RemoveContainer" containerID="1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638"
Nov 03 01:15:02.998884 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:02.998852    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638\": container with ID starting with 1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638 not found: ID does not exist" containerID="1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638"
Nov 03 01:15:02.999023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.998893    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638"} err="failed to get container status \"1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638\": rpc error: code = NotFound desc = could not find container \"1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638\": container with ID starting with 1262957b860c4a1fbc611a8918a965454ad8faeaac51bc14d6e30a088f128638 not found: ID does not exist"
Nov 03 01:15:02.999023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:02.998913    6000 scope.go:122] "RemoveContainer" containerID="ec3afc589b9cd969e9ff08319df1b57c82bf627959014727456e9a7dca924025"
Nov 03 01:15:03.024146 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.024123    6000 scope.go:122] "RemoveContainer" containerID="28e45991cfb9b2c8623cf3fc135dd980e7f415722d89ba9b9ec32ded431a4f95"
Nov 03 01:15:03.951054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.951022    6000 scope.go:122] "RemoveContainer" containerID="3557b094adc95afbbc4b27133c2be8c961a82c2dc3467bef1b61c7dbe2601bad"
Nov 03 01:15:03.951540 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:03.951378    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw_odh-ai-gateway-infra(508ab7fa-8fe3-4390-8a38-c40d94180c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e"
Nov 03 01:15:03.952643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.952613    6000 generic.go:356] "Generic (PLEG): container finished" podID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerID="c2c1a9f56142ba9d84783906e80b2d96a4b737da1725b0a2ae30504299ad3c7b" exitCode=60
Nov 03 01:15:03.952802 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.952685    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" event={"ID":"7551f80e-1fa7-4afd-acf5-f0364d517ac6","Type":"ContainerDied","Data":"c2c1a9f56142ba9d84783906e80b2d96a4b737da1725b0a2ae30504299ad3c7b"}
Nov 03 01:15:03.952802 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.952747    6000 scope.go:122] "RemoveContainer" containerID="834e095b6d45efef49eb0de2b8b42f17f96a624b0e8369bcc4b46e1a44e2152d"
Nov 03 01:15:03.953047 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.953018    6000 scope.go:122] "RemoveContainer" containerID="c2c1a9f56142ba9d84783906e80b2d96a4b737da1725b0a2ae30504299ad3c7b"
Nov 03 01:15:03.953335 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:03.953317    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp_odh-ai-gateway-infra(7551f80e-1fa7-4afd-acf5-f0364d517ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6"
Nov 03 01:15:03.954852 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.954834    6000 scope.go:122] "RemoveContainer" containerID="9825fc5c394484b4243bb89545dbdaaf9fba250bd2a8a3900ec33f1f69cd90e8"
Nov 03 01:15:03.955092 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:03.955074    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx_odh-ai-gateway-infra(ba47343d-7e39-4e7b-855a-04edb08f8ba1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1"
Nov 03 01:15:03.956429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:03.956409    6000 scope.go:122] "RemoveContainer" containerID="67c224f997f1faf3596a531cd37555828c8b2f8e6a49918a12787bd812b62094"
Nov 03 01:15:03.956669 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:03.956651    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7_odh-ai-gateway-infra(991686e6-13bc-41c3-a577-baf8686c1109)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" podUID="991686e6-13bc-41c3-a577-baf8686c1109"
Nov 03 01:15:04.964806 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:04.964777    6000 scope.go:122] "RemoveContainer" containerID="c2c1a9f56142ba9d84783906e80b2d96a4b737da1725b0a2ae30504299ad3c7b"
Nov 03 01:15:04.965219 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:04.965053    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp_odh-ai-gateway-infra(7551f80e-1fa7-4afd-acf5-f0364d517ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6"
Nov 03 01:15:06.403066 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:06.403030    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"]
Nov 03 01:15:10.601592 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:10.601558    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"]
Nov 03 01:15:11.404112 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:11.404075    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" podUID="7c7b20a3-0524-450e-b61b-c69543af4191" containerName="payload-processing" containerID="cri-o://819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e" gracePeriod=25
Nov 03 01:15:12.167739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.167661    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"]
Nov 03 01:15:12.171738 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.171716    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.199726 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.199695    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-099583-openshift-default-dockercfg-hbtbr\""
Nov 03 01:15:12.202200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.202170    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"]
Nov 03 01:15:12.231267 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231228    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231442 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231296    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231442 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231330    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231442 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231380    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231442 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231409    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231483    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231587    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231817 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231697    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.231817 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.231726    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqbv2\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-kube-api-access-jqbv2\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.332653 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332618    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.332841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332694    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.332841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332711    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqbv2\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-kube-api-access-jqbv2\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.332841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332752    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.332841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332768    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.332841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332782    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.332841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332822    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.333224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332846    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.333224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.332912    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.333224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.333102    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-certs\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.333409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.333219    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-data\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.333409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.333336    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.333409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.333356    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-credential-socket\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.333685 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.333664    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istiod-ca-cert\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.335419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.335388    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-envoy\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.335681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.335662    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-podinfo\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.352478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.352448    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-token\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.353081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.353057    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqbv2\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-kube-api-access-jqbv2\") pod \"e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") " pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.367353 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.367233    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"]
Nov 03 01:15:12.404117 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.404086    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" podUID="7c7b20a3-0524-450e-b61b-c69543af4191" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:12.485887 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.485691    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:12.818695 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.818666    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"]
Nov 03 01:15:12.821036 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:15:12.821010    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd96e482_f9fd_4acb_b4eb_195f6c98d526.slice/crio-68cccde5936a1e2eab63b1cf9679cadd35eb81c9202634b18379416b39e0c403 WatchSource:0}: Error finding container 68cccde5936a1e2eab63b1cf9679cadd35eb81c9202634b18379416b39e0c403: Status 404 returned error can't find the container with id 68cccde5936a1e2eab63b1cf9679cadd35eb81c9202634b18379416b39e0c403
Nov 03 01:15:12.826753 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.826716    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:15:12.826837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.826789    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:15:12.826837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:12.826818    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:15:13.017727 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:13.017692    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" event={"ID":"bd96e482-f9fd-4acb-b4eb-195f6c98d526","Type":"ContainerStarted","Data":"9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133"}
Nov 03 01:15:13.017952 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:13.017753    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" event={"ID":"bd96e482-f9fd-4acb-b4eb-195f6c98d526","Type":"ContainerStarted","Data":"68cccde5936a1e2eab63b1cf9679cadd35eb81c9202634b18379416b39e0c403"}
Nov 03 01:15:13.059501 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:13.059434    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podStartSLOduration=1.059412708 podStartE2EDuration="1.059412708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:13.058722982 +0000 UTC m=+4150.614075025" watchObservedRunningTime="2026-11-03 01:15:13.059412708 +0000 UTC m=+4150.614764733"
Nov 03 01:15:13.487147 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:13.487113    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:13.488763 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:13.488737    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:15:13.488816 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:13.488789    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 01:15:14.501823 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:14.501783    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:15:14.502472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:14.502437    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 01:15:14.559637 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:14.559609    6000 scope.go:122] "RemoveContainer" containerID="3557b094adc95afbbc4b27133c2be8c961a82c2dc3467bef1b61c7dbe2601bad"
Nov 03 01:15:15.014847 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.014594    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main" containerID="cri-o://cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268" gracePeriod=45
Nov 03 01:15:15.041073 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.041042    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" event={"ID":"508ab7fa-8fe3-4390-8a38-c40d94180c3e","Type":"ContainerStarted","Data":"40d8bb64607e0de79459dfa768999227c5fd3d31353f55367ad7a8d56b895fdb"}
Nov 03 01:15:15.082635 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.082538    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" podStartSLOduration=15.082521669 podStartE2EDuration="15.082521669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:15.080169757 +0000 UTC m=+4152.635521771" watchObservedRunningTime="2026-11-03 01:15:15.082521669 +0000 UTC m=+4152.637873692"
Nov 03 01:15:15.325639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.325611    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 01:15:15.373354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.373323    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-dshm\") pod \"02281a18-5373-45a1-89a6-25f6ca6e2911\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") "
Nov 03 01:15:15.373546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.373438    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02281a18-5373-45a1-89a6-25f6ca6e2911-tls-certs\" (UniqueName: \"kubernetes.io/secret/02281a18-5373-45a1-89a6-25f6ca6e2911-tls-certs\") pod \"02281a18-5373-45a1-89a6-25f6ca6e2911\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") "
Nov 03 01:15:15.373546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.373468    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-tmp-dir\") pod \"02281a18-5373-45a1-89a6-25f6ca6e2911\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") "
Nov 03 01:15:15.373678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.373567    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-model-cache\") pod \"02281a18-5373-45a1-89a6-25f6ca6e2911\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") "
Nov 03 01:15:15.373678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.373605    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-home\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-home\") pod \"02281a18-5373-45a1-89a6-25f6ca6e2911\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") "
Nov 03 01:15:15.373678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.373659    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02281a18-5373-45a1-89a6-25f6ca6e2911-kube-api-access-tvl9l\" (UniqueName: \"kubernetes.io/projected/02281a18-5373-45a1-89a6-25f6ca6e2911-kube-api-access-tvl9l\") pod \"02281a18-5373-45a1-89a6-25f6ca6e2911\" (UID: \"02281a18-5373-45a1-89a6-25f6ca6e2911\") "
Nov 03 01:15:15.373875 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.373838    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-tmp-dir" pod "02281a18-5373-45a1-89a6-25f6ca6e2911" (UID: "02281a18-5373-45a1-89a6-25f6ca6e2911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.374314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.374284    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-home" pod "02281a18-5373-45a1-89a6-25f6ca6e2911" (UID: "02281a18-5373-45a1-89a6-25f6ca6e2911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.374521 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.374436    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.374521 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.374459    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.375013 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.374987    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-model-cache" pod "02281a18-5373-45a1-89a6-25f6ca6e2911" (UID: "02281a18-5373-45a1-89a6-25f6ca6e2911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.418861 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.418825    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-dshm" pod "02281a18-5373-45a1-89a6-25f6ca6e2911" (UID: "02281a18-5373-45a1-89a6-25f6ca6e2911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:15.419232 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.419202    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02281a18-5373-45a1-89a6-25f6ca6e2911-tls-certs" pod "02281a18-5373-45a1-89a6-25f6ca6e2911" (UID: "02281a18-5373-45a1-89a6-25f6ca6e2911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:15.421768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.421732    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02281a18-5373-45a1-89a6-25f6ca6e2911-kube-api-access-tvl9l" pod "02281a18-5373-45a1-89a6-25f6ca6e2911" (UID: "02281a18-5373-45a1-89a6-25f6ca6e2911"). InnerVolumeSpecName "kube-api-access-tvl9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:15.476064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.476024    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.476064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.476068    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvl9l\" (UniqueName: \"kubernetes.io/projected/02281a18-5373-45a1-89a6-25f6ca6e2911-kube-api-access-tvl9l\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.476262 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.476084    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02281a18-5373-45a1-89a6-25f6ca6e2911-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.476262 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.476103    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02281a18-5373-45a1-89a6-25f6ca6e2911-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:15.500760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.497037    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:15:15.500760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.497095    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 01:15:15.734608 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:15.734522    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:15:16.049621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.049290    6000 generic.go:356] "Generic (PLEG): container finished" podID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerID="40d8bb64607e0de79459dfa768999227c5fd3d31353f55367ad7a8d56b895fdb" exitCode=60
Nov 03 01:15:16.049621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.049353    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" event={"ID":"508ab7fa-8fe3-4390-8a38-c40d94180c3e","Type":"ContainerDied","Data":"40d8bb64607e0de79459dfa768999227c5fd3d31353f55367ad7a8d56b895fdb"}
Nov 03 01:15:16.049621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.049419    6000 scope.go:122] "RemoveContainer" containerID="3557b094adc95afbbc4b27133c2be8c961a82c2dc3467bef1b61c7dbe2601bad"
Nov 03 01:15:16.049894 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.049853    6000 scope.go:122] "RemoveContainer" containerID="40d8bb64607e0de79459dfa768999227c5fd3d31353f55367ad7a8d56b895fdb"
Nov 03 01:15:16.050243 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:16.050223    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw_odh-ai-gateway-infra(508ab7fa-8fe3-4390-8a38-c40d94180c3e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e"
Nov 03 01:15:16.051498 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.051479    6000 generic.go:356] "Generic (PLEG): container finished" podID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerID="cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268" exitCode=0
Nov 03 01:15:16.051600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.051535    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"
Nov 03 01:15:16.051600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.051554    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" event={"ID":"02281a18-5373-45a1-89a6-25f6ca6e2911","Type":"ContainerDied","Data":"cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268"}
Nov 03 01:15:16.051600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.051588    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6" event={"ID":"02281a18-5373-45a1-89a6-25f6ca6e2911","Type":"ContainerDied","Data":"18c3709e82755bfb807da7cf0d4a43dfacda5acb3f1f4e8e7bd2837baa2a358a"}
Nov 03 01:15:16.076634 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.076602    6000 scope.go:122] "RemoveContainer" containerID="cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268"
Nov 03 01:15:16.086103 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.086079    6000 scope.go:122] "RemoveContainer" containerID="cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268"
Nov 03 01:15:16.107285 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:16.101444    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268\": container with ID starting with cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268 not found: ID does not exist" containerID="cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268"
Nov 03 01:15:16.107285 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.101500    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268"} err="failed to get container status \"cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268\": rpc error: code = NotFound desc = could not find container \"cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268\": container with ID starting with cf3421f0005956b88a10c7bb5e66158fe7360f0b3899668fc3ae844aa20ab268 not found: ID does not exist"
Nov 03 01:15:16.117221 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.117087    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"]
Nov 03 01:15:16.134224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.134190    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"]
Nov 03 01:15:16.140272 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.140227    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-4c61c8/premium-w2-4c61c8-kserve-695cf5d76f-xhtv6"]
Nov 03 01:15:16.491744 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.491702    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:15:16.492079 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.492043    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 01:15:16.559726 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.559694    6000 scope.go:122] "RemoveContainer" containerID="9825fc5c394484b4243bb89545dbdaaf9fba250bd2a8a3900ec33f1f69cd90e8"
Nov 03 01:15:16.567764 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:16.567737    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" path="/var/lib/kubelet/pods/02281a18-5373-45a1-89a6-25f6ca6e2911/volumes"
Nov 03 01:15:17.061652 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.061622    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" event={"ID":"ba47343d-7e39-4e7b-855a-04edb08f8ba1","Type":"ContainerStarted","Data":"53e5040d0507ae515c01bab541f03e9ae553ad851c3a08f5aa9f9a3e7fb5150f"}
Nov 03 01:15:17.156082 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.156006    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" podStartSLOduration=17.155982671 podStartE2EDuration="17.155982671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:17.153889278 +0000 UTC m=+4154.709241314" watchObservedRunningTime="2026-11-03 01:15:17.155982671 +0000 UTC m=+4154.711334729"
Nov 03 01:15:17.321506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.321422    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:17.429434 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.429401    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/508ab7fa-8fe3-4390-8a38-c40d94180c3e-kube-api-access-f5lv9\" (UniqueName: \"kubernetes.io/projected/508ab7fa-8fe3-4390-8a38-c40d94180c3e-kube-api-access-f5lv9\") pod \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") "
Nov 03 01:15:17.429660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.429622    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/508ab7fa-8fe3-4390-8a38-c40d94180c3e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ab7fa-8fe3-4390-8a38-c40d94180c3e-maas-api-service-ca\") pod \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\" (UID: \"508ab7fa-8fe3-4390-8a38-c40d94180c3e\") "
Nov 03 01:15:17.430042 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.430012    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/508ab7fa-8fe3-4390-8a38-c40d94180c3e-maas-api-service-ca" pod "508ab7fa-8fe3-4390-8a38-c40d94180c3e" (UID: "508ab7fa-8fe3-4390-8a38-c40d94180c3e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:17.432550 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.432516    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/508ab7fa-8fe3-4390-8a38-c40d94180c3e-kube-api-access-f5lv9" pod "508ab7fa-8fe3-4390-8a38-c40d94180c3e" (UID: "508ab7fa-8fe3-4390-8a38-c40d94180c3e"). InnerVolumeSpecName "kube-api-access-f5lv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:17.497283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.489655    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:15:17.497283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.489721    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 01:15:17.531701 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.531660    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5lv9\" (UniqueName: \"kubernetes.io/projected/508ab7fa-8fe3-4390-8a38-c40d94180c3e-kube-api-access-f5lv9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.531701 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.531695    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/508ab7fa-8fe3-4390-8a38-c40d94180c3e-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:17.559974 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.559934    6000 scope.go:122] "RemoveContainer" containerID="67c224f997f1faf3596a531cd37555828c8b2f8e6a49918a12787bd812b62094"
Nov 03 01:15:17.560183 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:17.560069    6000 scope.go:122] "RemoveContainer" containerID="c2c1a9f56142ba9d84783906e80b2d96a4b737da1725b0a2ae30504299ad3c7b"
Nov 03 01:15:18.070604 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.070580    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"
Nov 03 01:15:18.070604 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.070593    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw" event={"ID":"508ab7fa-8fe3-4390-8a38-c40d94180c3e","Type":"ContainerDied","Data":"ff1b5f805a87d423eb25f8859fb58e46c2fbc5292e126d55675bc46d22cf17f0"}
Nov 03 01:15:18.071114 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.070663    6000 scope.go:122] "RemoveContainer" containerID="40d8bb64607e0de79459dfa768999227c5fd3d31353f55367ad7a8d56b895fdb"
Nov 03 01:15:18.072872 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.072845    6000 generic.go:356] "Generic (PLEG): container finished" podID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerID="53e5040d0507ae515c01bab541f03e9ae553ad851c3a08f5aa9f9a3e7fb5150f" exitCode=60
Nov 03 01:15:18.072993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.072875    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" event={"ID":"ba47343d-7e39-4e7b-855a-04edb08f8ba1","Type":"ContainerDied","Data":"53e5040d0507ae515c01bab541f03e9ae553ad851c3a08f5aa9f9a3e7fb5150f"}
Nov 03 01:15:18.073275 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.073239    6000 scope.go:122] "RemoveContainer" containerID="53e5040d0507ae515c01bab541f03e9ae553ad851c3a08f5aa9f9a3e7fb5150f"
Nov 03 01:15:18.073563 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:18.073538    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx_odh-ai-gateway-infra(ba47343d-7e39-4e7b-855a-04edb08f8ba1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1"
Nov 03 01:15:18.075296 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.075241    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" event={"ID":"991686e6-13bc-41c3-a577-baf8686c1109","Type":"ContainerStarted","Data":"5df3014f3349aa69165173de97e65c338b68789c5e57314cfaa46435fe1139d4"}
Nov 03 01:15:18.082052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.082032    6000 scope.go:122] "RemoveContainer" containerID="9825fc5c394484b4243bb89545dbdaaf9fba250bd2a8a3900ec33f1f69cd90e8"
Nov 03 01:15:18.165697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.165668    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"]
Nov 03 01:15:18.211964 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.211871    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"]
Nov 03 01:15:18.241393 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.240660    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-b-0db47e-29894475-5w4gw"]
Nov 03 01:15:18.255744 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.255694    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" podStartSLOduration=18.255679914 podStartE2EDuration="18.255679914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:15:18.25346799 +0000 UTC m=+4155.808820015" watchObservedRunningTime="2026-11-03 01:15:18.255679914 +0000 UTC m=+4155.811031938"
Nov 03 01:15:18.495347 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.487658    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:15:18.495347 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.487721    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 01:15:18.567229 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:18.567193    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" path="/var/lib/kubelet/pods/508ab7fa-8fe3-4390-8a38-c40d94180c3e/volumes"
Nov 03 01:15:19.085130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.085039    6000 generic.go:356] "Generic (PLEG): container finished" podID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerID="761f65ec3b16037fb393c34618ce3250fb39b6df2068248b410157f3409338e9" exitCode=60
Nov 03 01:15:19.085559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.085129    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" event={"ID":"7551f80e-1fa7-4afd-acf5-f0364d517ac6","Type":"ContainerDied","Data":"761f65ec3b16037fb393c34618ce3250fb39b6df2068248b410157f3409338e9"}
Nov 03 01:15:19.085559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.085208    6000 scope.go:122] "RemoveContainer" containerID="c2c1a9f56142ba9d84783906e80b2d96a4b737da1725b0a2ae30504299ad3c7b"
Nov 03 01:15:19.085688 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.085669    6000 scope.go:122] "RemoveContainer" containerID="761f65ec3b16037fb393c34618ce3250fb39b6df2068248b410157f3409338e9"
Nov 03 01:15:19.086032 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:19.086007    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp_odh-ai-gateway-infra(7551f80e-1fa7-4afd-acf5-f0364d517ac6)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6"
Nov 03 01:15:19.105188 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.105150    6000 generic.go:356] "Generic (PLEG): container finished" podID="991686e6-13bc-41c3-a577-baf8686c1109" containerID="5df3014f3349aa69165173de97e65c338b68789c5e57314cfaa46435fe1139d4" exitCode=60
Nov 03 01:15:19.105372 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.105206    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" event={"ID":"991686e6-13bc-41c3-a577-baf8686c1109","Type":"ContainerDied","Data":"5df3014f3349aa69165173de97e65c338b68789c5e57314cfaa46435fe1139d4"}
Nov 03 01:15:19.105634 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.105613    6000 scope.go:122] "RemoveContainer" containerID="5df3014f3349aa69165173de97e65c338b68789c5e57314cfaa46435fe1139d4"
Nov 03 01:15:19.105924 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:19.105905    6000 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7_odh-ai-gateway-infra(991686e6-13bc-41c3-a577-baf8686c1109)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" podUID="991686e6-13bc-41c3-a577-baf8686c1109"
Nov 03 01:15:19.113965 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.113933    6000 scope.go:122] "RemoveContainer" containerID="67c224f997f1faf3596a531cd37555828c8b2f8e6a49918a12787bd812b62094"
Nov 03 01:15:19.265109 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.265066    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"]
Nov 03 01:15:19.272006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.271978    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:19.463185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.463147    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba47343d-7e39-4e7b-855a-04edb08f8ba1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba47343d-7e39-4e7b-855a-04edb08f8ba1-maas-api-service-ca\") pod \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") "
Nov 03 01:15:19.463434 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.463389    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba47343d-7e39-4e7b-855a-04edb08f8ba1-kube-api-access-btn9p\" (UniqueName: \"kubernetes.io/projected/ba47343d-7e39-4e7b-855a-04edb08f8ba1-kube-api-access-btn9p\") pod \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\" (UID: \"ba47343d-7e39-4e7b-855a-04edb08f8ba1\") "
Nov 03 01:15:19.463555 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.463531    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba47343d-7e39-4e7b-855a-04edb08f8ba1-maas-api-service-ca" pod "ba47343d-7e39-4e7b-855a-04edb08f8ba1" (UID: "ba47343d-7e39-4e7b-855a-04edb08f8ba1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:19.463834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.463815    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba47343d-7e39-4e7b-855a-04edb08f8ba1-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:19.465752 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.465733    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba47343d-7e39-4e7b-855a-04edb08f8ba1-kube-api-access-btn9p" pod "ba47343d-7e39-4e7b-855a-04edb08f8ba1" (UID: "ba47343d-7e39-4e7b-855a-04edb08f8ba1"). InnerVolumeSpecName "kube-api-access-btn9p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:19.487369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.487337    6000 patch_prober.go:29] interesting pod/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused" start-of-body=
Nov 03 01:15:19.487471 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.487397    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.45:15021/healthz/ready\": dial tcp 10.134.1.45:15021: connect: connection refused"
Nov 03 01:15:19.522238 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.522203    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"]
Nov 03 01:15:19.565580 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:19.565527    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btn9p\" (UniqueName: \"kubernetes.io/projected/ba47343d-7e39-4e7b-855a-04edb08f8ba1-kube-api-access-btn9p\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:19.901453 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:19.901416    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:15:20.127281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.119318    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx" event={"ID":"ba47343d-7e39-4e7b-855a-04edb08f8ba1","Type":"ContainerDied","Data":"ce5ed95ca5d3ab57f88cb363babbb00cc93560726a11eaa0d0d08e5d2bc6e7d0"}
Nov 03 01:15:20.127281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.122697    6000 scope.go:122] "RemoveContainer" containerID="53e5040d0507ae515c01bab541f03e9ae553ad851c3a08f5aa9f9a3e7fb5150f"
Nov 03 01:15:20.127281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.119247    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"
Nov 03 01:15:20.138162 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.138130    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"]
Nov 03 01:15:20.254473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.254444    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"]
Nov 03 01:15:20.257756 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.257732    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-5c109b-29894475-68tjx"]
Nov 03 01:15:20.327047 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.327019    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:20.378114 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.378083    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7551f80e-1fa7-4afd-acf5-f0364d517ac6-kube-api-access-jtl8k\" (UniqueName: \"kubernetes.io/projected/7551f80e-1fa7-4afd-acf5-f0364d517ac6-kube-api-access-jtl8k\") pod \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") "
Nov 03 01:15:20.378305 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.378221    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7551f80e-1fa7-4afd-acf5-f0364d517ac6-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7551f80e-1fa7-4afd-acf5-f0364d517ac6-maas-api-service-ca\") pod \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\" (UID: \"7551f80e-1fa7-4afd-acf5-f0364d517ac6\") "
Nov 03 01:15:20.379008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.378941    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7551f80e-1fa7-4afd-acf5-f0364d517ac6-maas-api-service-ca" pod "7551f80e-1fa7-4afd-acf5-f0364d517ac6" (UID: "7551f80e-1fa7-4afd-acf5-f0364d517ac6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:20.380598 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.380569    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7551f80e-1fa7-4afd-acf5-f0364d517ac6-kube-api-access-jtl8k" pod "7551f80e-1fa7-4afd-acf5-f0364d517ac6" (UID: "7551f80e-1fa7-4afd-acf5-f0364d517ac6"). InnerVolumeSpecName "kube-api-access-jtl8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:20.398391 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.398371    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:20.479743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.479654    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/991686e6-13bc-41c3-a577-baf8686c1109-kube-api-access-9nm2f\" (UniqueName: \"kubernetes.io/projected/991686e6-13bc-41c3-a577-baf8686c1109-kube-api-access-9nm2f\") pod \"991686e6-13bc-41c3-a577-baf8686c1109\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") "
Nov 03 01:15:20.479743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.479703    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/991686e6-13bc-41c3-a577-baf8686c1109-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991686e6-13bc-41c3-a577-baf8686c1109-maas-api-service-ca\") pod \"991686e6-13bc-41c3-a577-baf8686c1109\" (UID: \"991686e6-13bc-41c3-a577-baf8686c1109\") "
Nov 03 01:15:20.480207 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.480179    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/991686e6-13bc-41c3-a577-baf8686c1109-maas-api-service-ca" pod "991686e6-13bc-41c3-a577-baf8686c1109" (UID: "991686e6-13bc-41c3-a577-baf8686c1109"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:20.480334 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.480208    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7551f80e-1fa7-4afd-acf5-f0364d517ac6-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:20.480334 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.480230    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jtl8k\" (UniqueName: \"kubernetes.io/projected/7551f80e-1fa7-4afd-acf5-f0364d517ac6-kube-api-access-jtl8k\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:20.482105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.482083    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/991686e6-13bc-41c3-a577-baf8686c1109-kube-api-access-9nm2f" pod "991686e6-13bc-41c3-a577-baf8686c1109" (UID: "991686e6-13bc-41c3-a577-baf8686c1109"). InnerVolumeSpecName "kube-api-access-9nm2f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:20.491612 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.491583    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:20.491881 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.491863    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:20.492516 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.492498    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:15:20.568113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.568082    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" path="/var/lib/kubelet/pods/ba47343d-7e39-4e7b-855a-04edb08f8ba1/volumes"
Nov 03 01:15:20.585295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.581298    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nm2f\" (UniqueName: \"kubernetes.io/projected/991686e6-13bc-41c3-a577-baf8686c1109-kube-api-access-9nm2f\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:20.585295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:20.581331    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/991686e6-13bc-41c3-a577-baf8686c1109-maas-api-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:21.128328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.128298    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"
Nov 03 01:15:21.128823 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.128302    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp" event={"ID":"7551f80e-1fa7-4afd-acf5-f0364d517ac6","Type":"ContainerDied","Data":"2716243c83ea20bad34f0187cddc06a1e29139426caa567444cf8d66bb1b497b"}
Nov 03 01:15:21.128823 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.128438    6000 scope.go:122] "RemoveContainer" containerID="761f65ec3b16037fb393c34618ce3250fb39b6df2068248b410157f3409338e9"
Nov 03 01:15:21.131767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.131749    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"
Nov 03 01:15:21.131981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.131782    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7" event={"ID":"991686e6-13bc-41c3-a577-baf8686c1109","Type":"ContainerDied","Data":"e18c02f794b5d9df7bd0c93deb84ab7e032ff2768e2480bd023351d898884d52"}
Nov 03 01:15:21.139346 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.139329    6000 scope.go:122] "RemoveContainer" containerID="5df3014f3349aa69165173de97e65c338b68789c5e57314cfaa46435fe1139d4"
Nov 03 01:15:21.182191 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.182164    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"]
Nov 03 01:15:21.195201 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.195173    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-99bdb8-29894475-r27wp"]
Nov 03 01:15:21.228086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.228041    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"]
Nov 03 01:15:21.232311 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:21.232288    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-9908e8-29894475-26xc7"]
Nov 03 01:15:22.405349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:22.405312    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" podUID="7c7b20a3-0524-450e-b61b-c69543af4191" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:22.568048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:22.568014    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" path="/var/lib/kubelet/pods/7551f80e-1fa7-4afd-acf5-f0364d517ac6/volumes"
Nov 03 01:15:22.568533 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:22.568518    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="991686e6-13bc-41c3-a577-baf8686c1109" path="/var/lib/kubelet/pods/991686e6-13bc-41c3-a577-baf8686c1109/volumes"
Nov 03 01:15:24.522800 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:24.522697    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" podUID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" containerName="payload-processing" containerID="cri-o://f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613" gracePeriod=25
Nov 03 01:15:25.626345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.626309    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq" podUID="f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" containerName="main" containerID="cri-o://9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681" gracePeriod=45
Nov 03 01:15:25.904322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.904295    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:15:25.941385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941350    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-home\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-home\") pod \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") "
Nov 03 01:15:25.941553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941464    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tmp-dir\") pod \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") "
Nov 03 01:15:25.941553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941496    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tls-certs\") pod \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") "
Nov 03 01:15:25.941649 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941609    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-home" pod "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" (UID: "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:25.941649 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941629    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-model-cache\") pod \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") "
Nov 03 01:15:25.941768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941713    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-dshm\") pod \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") "
Nov 03 01:15:25.941768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941745    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tmp-dir" pod "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" (UID: "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:25.941768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941764    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-kube-api-access-r77zs\" (UniqueName: \"kubernetes.io/projected/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-kube-api-access-r77zs\") pod \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\" (UID: \"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8\") "
Nov 03 01:15:25.941933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.941910    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-model-cache" pod "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" (UID: "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:25.942380 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.942294    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:25.942380 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.942324    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:25.942725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.942679    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:25.944947 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.944649    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tls-certs" pod "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" (UID: "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:25.946335 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.946313    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-dshm" pod "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" (UID: "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:25.946470 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:25.946353    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-kube-api-access-r77zs" pod "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" (UID: "f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8"). InnerVolumeSpecName "kube-api-access-r77zs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:26.043782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.043744    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:26.043782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.043777    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:26.043782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.043786    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r77zs\" (UniqueName: \"kubernetes.io/projected/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8-kube-api-access-r77zs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:26.163049 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.162960    6000 generic.go:356] "Generic (PLEG): container finished" podID="f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" containerID="9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681" exitCode=0
Nov 03 01:15:26.163049 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.163004    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"
Nov 03 01:15:26.163049 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.163018    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq" event={"ID":"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8","Type":"ContainerDied","Data":"9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681"}
Nov 03 01:15:26.163049 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.163049    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq" event={"ID":"f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8","Type":"ContainerDied","Data":"de287d236c0933129722c5a6e24f8b99ca160e38ec708fb60fc9e7c06df5efb4"}
Nov 03 01:15:26.163378 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.163065    6000 scope.go:122] "RemoveContainer" containerID="9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681"
Nov 03 01:15:26.173298 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.173282    6000 scope.go:122] "RemoveContainer" containerID="9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681"
Nov 03 01:15:26.173554 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:26.173535    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681\": container with ID starting with 9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681 not found: ID does not exist" containerID="9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681"
Nov 03 01:15:26.173599 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.173561    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681"} err="failed to get container status \"9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681\": rpc error: code = NotFound desc = could not find container \"9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681\": container with ID starting with 9b13df817b056e05acc5c4b0df1c2c0b5a0c50043a9f812e6c3c970071b57681 not found: ID does not exist"
Nov 03 01:15:26.210644 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.210622    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"]
Nov 03 01:15:26.221671 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.221643    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-9908e8/sub-test-model-9908e8-kserve-7fdf6c648c-mk5nq"]
Nov 03 01:15:26.566931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:26.566849    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" path="/var/lib/kubelet/pods/f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8/volumes"
Nov 03 01:15:27.391388 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.391345    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m" podUID="f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" containerName="main" containerID="cri-o://99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26" gracePeriod=45
Nov 03 01:15:27.678322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.678286    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:15:27.767415 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767384    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-dshm\") pod \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") "
Nov 03 01:15:27.767581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767453    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-model-cache\") pod \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") "
Nov 03 01:15:27.767581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767495    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-home\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-home\") pod \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") "
Nov 03 01:15:27.767581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767525    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tls-certs\") pod \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") "
Nov 03 01:15:27.767581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767547    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tmp-dir\") pod \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") "
Nov 03 01:15:27.767842 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767714    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-kube-api-access-dckns\" (UniqueName: \"kubernetes.io/projected/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-kube-api-access-dckns\") pod \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\" (UID: \"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7\") "
Nov 03 01:15:27.767842 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767784    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-model-cache" pod "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" (UID: "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:27.767842 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767777    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-home" pod "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" (UID: "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:27.768002 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.767866    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tmp-dir" pod "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" (UID: "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:27.768177 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.768154    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:27.768177 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.768176    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:27.768383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.768187    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:27.769889 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.769868    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-dshm" pod "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" (UID: "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:27.769971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.769916    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-kube-api-access-dckns" pod "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" (UID: "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7"). InnerVolumeSpecName "kube-api-access-dckns". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:27.770008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.769983    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tls-certs" pod "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" (UID: "f985caf0-f2c4-4f84-b3e8-b47f4a5368e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:27.869461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.869425    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:27.869461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.869453    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:27.869461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:27.869465    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dckns\" (UniqueName: \"kubernetes.io/projected/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7-kube-api-access-dckns\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:28.177558 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.177521    6000 generic.go:356] "Generic (PLEG): container finished" podID="f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" containerID="99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26" exitCode=0
Nov 03 01:15:28.177736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.177566    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"
Nov 03 01:15:28.177736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.177567    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m" event={"ID":"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7","Type":"ContainerDied","Data":"99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26"}
Nov 03 01:15:28.177736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.177689    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m" event={"ID":"f985caf0-f2c4-4f84-b3e8-b47f4a5368e7","Type":"ContainerDied","Data":"35079960df13188284e5e9966f092ea9ae56860fa3be17d6e82824ce3dc43fa1"}
Nov 03 01:15:28.177736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.177710    6000 scope.go:122] "RemoveContainer" containerID="99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26"
Nov 03 01:15:28.188364 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.188345    6000 scope.go:122] "RemoveContainer" containerID="99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26"
Nov 03 01:15:28.188607 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:28.188586    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26\": container with ID starting with 99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26 not found: ID does not exist" containerID="99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26"
Nov 03 01:15:28.188681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.188614    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26"} err="failed to get container status \"99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26\": rpc error: code = NotFound desc = could not find container \"99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26\": container with ID starting with 99e0da377f236e6186554fa22b10d9a6db2e05ebbaefdf2ad9330e2692571b26 not found: ID does not exist"
Nov 03 01:15:28.225403 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.225375    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"]
Nov 03 01:15:28.236854 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.236831    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-20109d/sub-test-model-20109d-kserve-74ff4cd6fc-dgl9m"]
Nov 03 01:15:28.566969 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.566876    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" path="/var/lib/kubelet/pods/f985caf0-f2c4-4f84-b3e8-b47f4a5368e7/volumes"
Nov 03 01:15:28.716661 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:28.716628    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" podUID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:31.667399 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:31.667336    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 01:15:31.815866 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:31.815833    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c7b20a3-0524-450e-b61b-c69543af4191-kube-api-access-qzwtp\" (UniqueName: \"kubernetes.io/projected/7c7b20a3-0524-450e-b61b-c69543af4191-kube-api-access-qzwtp\") pod \"7c7b20a3-0524-450e-b61b-c69543af4191\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") "
Nov 03 01:15:31.816020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:31.815918    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c7b20a3-0524-450e-b61b-c69543af4191-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7b20a3-0524-450e-b61b-c69543af4191-plugins-config-volume\") pod \"7c7b20a3-0524-450e-b61b-c69543af4191\" (UID: \"7c7b20a3-0524-450e-b61b-c69543af4191\") "
Nov 03 01:15:31.816334 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:31.816304    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c7b20a3-0524-450e-b61b-c69543af4191-plugins-config-volume" pod "7c7b20a3-0524-450e-b61b-c69543af4191" (UID: "7c7b20a3-0524-450e-b61b-c69543af4191"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:31.818099 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:31.818075    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c7b20a3-0524-450e-b61b-c69543af4191-kube-api-access-qzwtp" pod "7c7b20a3-0524-450e-b61b-c69543af4191" (UID: "7c7b20a3-0524-450e-b61b-c69543af4191"). InnerVolumeSpecName "kube-api-access-qzwtp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:31.917882 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:31.917798    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzwtp\" (UniqueName: \"kubernetes.io/projected/7c7b20a3-0524-450e-b61b-c69543af4191-kube-api-access-qzwtp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:31.917882 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:31.917824    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7c7b20a3-0524-450e-b61b-c69543af4191-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:32.175790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.175701    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"]
Nov 03 01:15:32.205504 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.205466    6000 generic.go:356] "Generic (PLEG): container finished" podID="7c7b20a3-0524-450e-b61b-c69543af4191" containerID="819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e" exitCode=0
Nov 03 01:15:32.205677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.205521    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"
Nov 03 01:15:32.205677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.205551    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" event={"ID":"7c7b20a3-0524-450e-b61b-c69543af4191","Type":"ContainerDied","Data":"819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e"}
Nov 03 01:15:32.205677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.205605    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w" event={"ID":"7c7b20a3-0524-450e-b61b-c69543af4191","Type":"ContainerDied","Data":"7e7f18e2f4a30c54bdf2adf5fed18f0bd96f3686f1e23fc86f4b6f82c1d1c787"}
Nov 03 01:15:32.205677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.205635    6000 scope.go:122] "RemoveContainer" containerID="819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e"
Nov 03 01:15:32.218993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.218974    6000 scope.go:122] "RemoveContainer" containerID="819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e"
Nov 03 01:15:32.219349 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:32.219296    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e\": container with ID starting with 819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e not found: ID does not exist" containerID="819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e"
Nov 03 01:15:32.219349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.219328    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e"} err="failed to get container status \"819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e\": rpc error: code = NotFound desc = could not find container \"819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e\": container with ID starting with 819e323622c8e4808e366e7951cb76278a6c94799bbd3a604485a73a4fb61a1e not found: ID does not exist"
Nov 03 01:15:32.277979 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.277937    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"]
Nov 03 01:15:32.283190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.283147    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-4c61c8-577656bb65-k2z2w"]
Nov 03 01:15:32.572818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:32.572741    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c7b20a3-0524-450e-b61b-c69543af4191" path="/var/lib/kubelet/pods/7c7b20a3-0524-450e-b61b-c69543af4191/volumes"
Nov 03 01:15:35.509289 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.509235    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"]
Nov 03 01:15:35.509760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.509450    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv" podUID="7fa21b75-6104-431b-ba39-a56035f517ac" containerName="maas-api" containerID="cri-o://aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4" gracePeriod=30
Nov 03 01:15:35.767359 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.767289    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 01:15:35.866363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.866333    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa21b75-6104-431b-ba39-a56035f517ac-kube-api-access-q5zg9\" (UniqueName: \"kubernetes.io/projected/7fa21b75-6104-431b-ba39-a56035f517ac-kube-api-access-q5zg9\") pod \"7fa21b75-6104-431b-ba39-a56035f517ac\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") "
Nov 03 01:15:35.866541 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.866431    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa21b75-6104-431b-ba39-a56035f517ac-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa21b75-6104-431b-ba39-a56035f517ac-openshift-service-ca\") pod \"7fa21b75-6104-431b-ba39-a56035f517ac\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") "
Nov 03 01:15:35.866541 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.866483    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-metrics-certs\") pod \"7fa21b75-6104-431b-ba39-a56035f517ac\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") "
Nov 03 01:15:35.866626 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.866562    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-maas-api-tls\") pod \"7fa21b75-6104-431b-ba39-a56035f517ac\" (UID: \"7fa21b75-6104-431b-ba39-a56035f517ac\") "
Nov 03 01:15:35.866839 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.866798    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa21b75-6104-431b-ba39-a56035f517ac-openshift-service-ca" pod "7fa21b75-6104-431b-ba39-a56035f517ac" (UID: "7fa21b75-6104-431b-ba39-a56035f517ac"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:35.867011 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.866988    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fa21b75-6104-431b-ba39-a56035f517ac-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:35.868844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.868816    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa21b75-6104-431b-ba39-a56035f517ac-kube-api-access-q5zg9" pod "7fa21b75-6104-431b-ba39-a56035f517ac" (UID: "7fa21b75-6104-431b-ba39-a56035f517ac"). InnerVolumeSpecName "kube-api-access-q5zg9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:35.868936 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.868867    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-maas-api-tls" pod "7fa21b75-6104-431b-ba39-a56035f517ac" (UID: "7fa21b75-6104-431b-ba39-a56035f517ac"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:35.868936 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.868881    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-metrics-certs" pod "7fa21b75-6104-431b-ba39-a56035f517ac" (UID: "7fa21b75-6104-431b-ba39-a56035f517ac"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:35.968093 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.968056    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5zg9\" (UniqueName: \"kubernetes.io/projected/7fa21b75-6104-431b-ba39-a56035f517ac-kube-api-access-q5zg9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:35.968093 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.968090    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:35.968357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:35.968104    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7fa21b75-6104-431b-ba39-a56035f517ac-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:36.256321 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.256281    6000 generic.go:356] "Generic (PLEG): container finished" podID="7fa21b75-6104-431b-ba39-a56035f517ac" containerID="aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4" exitCode=0
Nov 03 01:15:36.256480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.256331    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"
Nov 03 01:15:36.256480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.256355    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv" event={"ID":"7fa21b75-6104-431b-ba39-a56035f517ac","Type":"ContainerDied","Data":"aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4"}
Nov 03 01:15:36.256480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.256414    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv" event={"ID":"7fa21b75-6104-431b-ba39-a56035f517ac","Type":"ContainerDied","Data":"88bf1abd02f163c8524355e8bbf954c6e87f70e811a516af250276fde98e0ca5"}
Nov 03 01:15:36.256480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.256437    6000 scope.go:122] "RemoveContainer" containerID="aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4"
Nov 03 01:15:36.266594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.266576    6000 scope.go:122] "RemoveContainer" containerID="aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4"
Nov 03 01:15:36.266893 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:36.266866    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4\": container with ID starting with aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4 not found: ID does not exist" containerID="aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4"
Nov 03 01:15:36.266990 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.266897    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4"} err="failed to get container status \"aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4\": rpc error: code = NotFound desc = could not find container \"aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4\": container with ID starting with aa641fa25b8923ed4631417bdd7207613ebb62f66c69db12fbc85df48fe56fc4 not found: ID does not exist"
Nov 03 01:15:36.306417 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.306388    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"]
Nov 03 01:15:36.311776 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.311752    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-4c61c8-55dd97f98b-rvhzv"]
Nov 03 01:15:36.567742 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:36.567660    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa21b75-6104-431b-ba39-a56035f517ac" path="/var/lib/kubelet/pods/7fa21b75-6104-431b-ba39-a56035f517ac/volumes"
Nov 03 01:15:38.716369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:38.716328    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" podUID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:15:42.904052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:42.904009    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"]
Nov 03 01:15:43.438536 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:43.438497    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:15:44.792836 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:44.792775    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 01:15:44.876530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:44.876488    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-plugins-config-volume\") pod \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") "
Nov 03 01:15:44.876742 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:44.876584    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-kube-api-access-bc225\") pod \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\" (UID: \"4cbb9852-0c50-49c4-bb1b-631b6c394e8b\") "
Nov 03 01:15:44.876854 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:44.876830    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-plugins-config-volume" pod "4cbb9852-0c50-49c4-bb1b-631b6c394e8b" (UID: "4cbb9852-0c50-49c4-bb1b-631b6c394e8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:44.877033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:44.877017    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:44.878974 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:44.878942    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-kube-api-access-bc225" pod "4cbb9852-0c50-49c4-bb1b-631b6c394e8b" (UID: "4cbb9852-0c50-49c4-bb1b-631b6c394e8b"). InnerVolumeSpecName "kube-api-access-bc225". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:44.978119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:44.978090    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc225\" (UniqueName: \"kubernetes.io/projected/4cbb9852-0c50-49c4-bb1b-631b6c394e8b-kube-api-access-bc225\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:45.312885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.312846    6000 generic.go:356] "Generic (PLEG): container finished" podID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" containerID="f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613" exitCode=0
Nov 03 01:15:45.313081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.312910    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"
Nov 03 01:15:45.313081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.312938    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" event={"ID":"4cbb9852-0c50-49c4-bb1b-631b6c394e8b","Type":"ContainerDied","Data":"f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613"}
Nov 03 01:15:45.313081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.313019    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp" event={"ID":"4cbb9852-0c50-49c4-bb1b-631b6c394e8b","Type":"ContainerDied","Data":"f15e69f67b10ed03b552b9f1507c4112132aff148c704e32d631347e977b2976"}
Nov 03 01:15:45.313081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.313043    6000 scope.go:122] "RemoveContainer" containerID="f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613"
Nov 03 01:15:45.323319 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.323296    6000 scope.go:122] "RemoveContainer" containerID="f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613"
Nov 03 01:15:45.323596 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:45.323572    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613\": container with ID starting with f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613 not found: ID does not exist" containerID="f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613"
Nov 03 01:15:45.323651 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.323605    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613"} err="failed to get container status \"f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613\": rpc error: code = NotFound desc = could not find container \"f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613\": container with ID starting with f6860ad0cae6f28a2bd264ec2451a11d1b4cc95674d2c4fbe379997e41fc6613 not found: ID does not exist"
Nov 03 01:15:45.361597 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.361567    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"]
Nov 03 01:15:45.372889 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:45.372858    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-850433-7c84b7db7c-58vjp"]
Nov 03 01:15:46.567572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:46.567541    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" path="/var/lib/kubelet/pods/4cbb9852-0c50-49c4-bb1b-631b6c394e8b/volumes"
Nov 03 01:15:46.732034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:46.731996    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"]
Nov 03 01:15:47.134985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.134934    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"]
Nov 03 01:15:47.136009 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.135170    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy" containerID="cri-o://8f724a75f11f39072edeca649bc0d55094f6c10488d2dbcd3b428c3094f69303" gracePeriod=30
Nov 03 01:15:47.200377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.200337    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt" podUID="54d2cebd-68c0-4b15-ac8e-5088762e5cdf" containerName="main" containerID="cri-o://04a636572fa2f9daa30e0d8b02028198206cebd0a967660499c93f482ce43006" gracePeriod=45
Nov 03 01:15:47.330936 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.330903    6000 generic.go:356] "Generic (PLEG): container finished" podID="54d2cebd-68c0-4b15-ac8e-5088762e5cdf" containerID="04a636572fa2f9daa30e0d8b02028198206cebd0a967660499c93f482ce43006" exitCode=0
Nov 03 01:15:47.331129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.330987    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt" event={"ID":"54d2cebd-68c0-4b15-ac8e-5088762e5cdf","Type":"ContainerDied","Data":"04a636572fa2f9daa30e0d8b02028198206cebd0a967660499c93f482ce43006"}
Nov 03 01:15:47.481497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.481465    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:15:47.507663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.507626    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-kube-api-access-sfnpx\" (UniqueName: \"kubernetes.io/projected/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-kube-api-access-sfnpx\") pod \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") "
Nov 03 01:15:47.507853 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.507681    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-dshm\") pod \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") "
Nov 03 01:15:47.507853 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.507802    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-model-cache\") pod \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") "
Nov 03 01:15:47.507985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.507868    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-home\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-home\") pod \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") "
Nov 03 01:15:47.507985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.507943    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tmp-dir\") pod \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") "
Nov 03 01:15:47.508121 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.507996    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tls-certs\" (UniqueName: \"kubernetes.io/secret/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tls-certs\") pod \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\" (UID: \"54d2cebd-68c0-4b15-ac8e-5088762e5cdf\") "
Nov 03 01:15:47.508121 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.508086    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-model-cache" pod "54d2cebd-68c0-4b15-ac8e-5088762e5cdf" (UID: "54d2cebd-68c0-4b15-ac8e-5088762e5cdf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:47.508230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.508116    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-home" pod "54d2cebd-68c0-4b15-ac8e-5088762e5cdf" (UID: "54d2cebd-68c0-4b15-ac8e-5088762e5cdf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:47.508472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.508447    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:47.508588 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.508474    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:47.508588 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.508511    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tmp-dir" pod "54d2cebd-68c0-4b15-ac8e-5088762e5cdf" (UID: "54d2cebd-68c0-4b15-ac8e-5088762e5cdf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:47.510023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.510003    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-dshm" pod "54d2cebd-68c0-4b15-ac8e-5088762e5cdf" (UID: "54d2cebd-68c0-4b15-ac8e-5088762e5cdf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:47.510122 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.510108    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-kube-api-access-sfnpx" pod "54d2cebd-68c0-4b15-ac8e-5088762e5cdf" (UID: "54d2cebd-68c0-4b15-ac8e-5088762e5cdf"). InnerVolumeSpecName "kube-api-access-sfnpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:47.510295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.510244    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tls-certs" pod "54d2cebd-68c0-4b15-ac8e-5088762e5cdf" (UID: "54d2cebd-68c0-4b15-ac8e-5088762e5cdf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:47.610077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.610039    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:47.610642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.610078    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfnpx\" (UniqueName: \"kubernetes.io/projected/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-kube-api-access-sfnpx\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:47.610642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.610104    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:47.610642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:47.610114    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54d2cebd-68c0-4b15-ac8e-5088762e5cdf-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:48.338412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:48.338380    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"
Nov 03 01:15:48.338412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:48.338405    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt" event={"ID":"54d2cebd-68c0-4b15-ac8e-5088762e5cdf","Type":"ContainerDied","Data":"0d75f92da6a4c400028c52e88cc3e63281026dd549bcf7e3637b506656df0558"}
Nov 03 01:15:48.338709 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:48.338456    6000 scope.go:122] "RemoveContainer" containerID="04a636572fa2f9daa30e0d8b02028198206cebd0a967660499c93f482ce43006"
Nov 03 01:15:48.386716 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:48.386689    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"]
Nov 03 01:15:48.397772 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:48.397748    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-099583/sub-test-model-099583-kserve-596b4695f7-jjtmt"]
Nov 03 01:15:48.566357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:48.566321    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54d2cebd-68c0-4b15-ac8e-5088762e5cdf" path="/var/lib/kubelet/pods/54d2cebd-68c0-4b15-ac8e-5088762e5cdf/volumes"
Nov 03 01:15:49.896997 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:49.896962    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:15:51.732569 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:51.732529    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6" podUID="c37bf3b6-0b50-4f34-837b-15368f5d6de7" containerName="payload-pre-processing" containerID="cri-o://883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991" gracePeriod=25
Nov 03 01:15:51.781072 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:15:51.781044    6000 logging.go:55] [core] [Channel #20273 SubChannel #20274]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.191:50052", ServerName: "10.134.0.191:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.191:50052: connect: connection refused"
Nov 03 01:15:51.996067 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:51.996008    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 01:15:52.060926 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.059639    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c37bf3b6-0b50-4f34-837b-15368f5d6de7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37bf3b6-0b50-4f34-837b-15368f5d6de7-plugins-config-volume\") pod \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") "
Nov 03 01:15:52.060926 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.059747    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c37bf3b6-0b50-4f34-837b-15368f5d6de7-kube-api-access-p7cxd\" (UniqueName: \"kubernetes.io/projected/c37bf3b6-0b50-4f34-837b-15368f5d6de7-kube-api-access-p7cxd\") pod \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\" (UID: \"c37bf3b6-0b50-4f34-837b-15368f5d6de7\") "
Nov 03 01:15:52.061684 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.061647    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c37bf3b6-0b50-4f34-837b-15368f5d6de7-plugins-config-volume" pod "c37bf3b6-0b50-4f34-837b-15368f5d6de7" (UID: "c37bf3b6-0b50-4f34-837b-15368f5d6de7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:52.068868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.068829    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c37bf3b6-0b50-4f34-837b-15368f5d6de7-kube-api-access-p7cxd" pod "c37bf3b6-0b50-4f34-837b-15368f5d6de7" (UID: "c37bf3b6-0b50-4f34-837b-15368f5d6de7"). InnerVolumeSpecName "kube-api-access-p7cxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:52.161918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.161891    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c37bf3b6-0b50-4f34-837b-15368f5d6de7-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.162032 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.161921    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7cxd\" (UniqueName: \"kubernetes.io/projected/c37bf3b6-0b50-4f34-837b-15368f5d6de7-kube-api-access-p7cxd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.364533 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.364493    6000 generic.go:356] "Generic (PLEG): container finished" podID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerID="8f724a75f11f39072edeca649bc0d55094f6c10488d2dbcd3b428c3094f69303" exitCode=0
Nov 03 01:15:52.364730 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.364595    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerDied","Data":"8f724a75f11f39072edeca649bc0d55094f6c10488d2dbcd3b428c3094f69303"}
Nov 03 01:15:52.364730 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.364657    6000 scope.go:122] "RemoveContainer" containerID="e67627834272cfb28ec16b5acf081b32cac27572eaf42b2c2bf6d772ec71b47e"
Nov 03 01:15:52.366233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.366209    6000 generic.go:356] "Generic (PLEG): container finished" podID="c37bf3b6-0b50-4f34-837b-15368f5d6de7" containerID="883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991" exitCode=0
Nov 03 01:15:52.366401 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.366272    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"
Nov 03 01:15:52.366401 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.366293    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6" event={"ID":"c37bf3b6-0b50-4f34-837b-15368f5d6de7","Type":"ContainerDied","Data":"883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991"}
Nov 03 01:15:52.366401 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.366360    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6" event={"ID":"c37bf3b6-0b50-4f34-837b-15368f5d6de7","Type":"ContainerDied","Data":"8bf75618c816905595a1643561ebc7f5fc43612ed607a3976b50af77cf44c87d"}
Nov 03 01:15:52.378703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.378681    6000 scope.go:122] "RemoveContainer" containerID="883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991"
Nov 03 01:15:52.388267 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.388228    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 01:15:52.389030 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.389009    6000 scope.go:122] "RemoveContainer" containerID="883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991"
Nov 03 01:15:52.389307 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:52.389288    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991\": container with ID starting with 883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991 not found: ID does not exist" containerID="883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991"
Nov 03 01:15:52.389363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.389318    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991"} err="failed to get container status \"883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991\": rpc error: code = NotFound desc = could not find container \"883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991\": container with ID starting with 883729e57f844e87038a625430bfcd7f0b2be9bbf1f02af003698dd21a748991 not found: ID does not exist"
Nov 03 01:15:52.417183 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.417158    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"]
Nov 03 01:15:52.428889 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.428865    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6"]
Nov 03 01:15:52.464823 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.464794    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-kube-api-access-4kd86\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-kube-api-access-4kd86\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.464953 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.464904    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-credential-socket\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.465023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.464951    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-certs\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.465023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.464989    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-data\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.465736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.465242    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-data" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:52.465736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.465269    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-credential-socket" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:52.465736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.465082    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-podinfo\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.465968 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.465766    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istiod-ca-cert\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.465968 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.465875    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-envoy\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.465968 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.465918    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-socket\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.465968 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.465949    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-token\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-token\") pod \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\" (UID: \"e4d39152-3fe9-47bd-b166-9a84ac7a5f98\") "
Nov 03 01:15:52.466986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.466456    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.466986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.466482    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.466986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.466587    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-socket" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:52.466986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.466958    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-certs" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:52.468815 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.468789    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-podinfo" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:15:52.468924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.468895    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istiod-ca-cert" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:52.469032 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.469012    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-envoy" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:52.469133 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.469094    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-kube-api-access-4kd86" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "kube-api-access-4kd86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:52.469562 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.469537    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-token" pod "e4d39152-3fe9-47bd-b166-9a84ac7a5f98" (UID: "e4d39152-3fe9-47bd-b166-9a84ac7a5f98"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:52.566399 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.566319    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c37bf3b6-0b50-4f34-837b-15368f5d6de7" path="/var/lib/kubelet/pods/c37bf3b6-0b50-4f34-837b-15368f5d6de7/volumes"
Nov 03 01:15:52.567463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.567432    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kd86\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-kube-api-access-4kd86\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.567463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.567459    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.567463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.567469    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.567729 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.567477    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.567729 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.567486    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.567729 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.567496    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.567729 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.567505    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4d39152-3fe9-47bd-b166-9a84ac7a5f98-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:52.781666 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:52.781628    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-ac338d-5cfd9848b7-qt6x6" podUID="c37bf3b6-0b50-4f34-837b-15368f5d6de7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.191:50052\" within 1s: context deadline exceeded"
Nov 03 01:15:53.373170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:53.373124    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8" event={"ID":"e4d39152-3fe9-47bd-b166-9a84ac7a5f98","Type":"ContainerDied","Data":"c3b296fbbadba7c0b1c7e710db3630781fed7f012baf85d376ae31e197982658"}
Nov 03 01:15:53.373170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:53.373154    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"
Nov 03 01:15:53.373488 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:53.373205    6000 scope.go:122] "RemoveContainer" containerID="8f724a75f11f39072edeca649bc0d55094f6c10488d2dbcd3b428c3094f69303"
Nov 03 01:15:53.433721 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:53.433686    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"]
Nov 03 01:15:53.445076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:53.445052    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-4c61c8-openshift-default-64bcc6f947-zzcl8"]
Nov 03 01:15:54.566738 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:54.566697    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" path="/var/lib/kubelet/pods/e4d39152-3fe9-47bd-b166-9a84ac7a5f98/volumes"
Nov 03 01:15:57.503368 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:57.503334    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:15:57.925783 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:57.925741    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx" podUID="66432f27-b3ec-443d-a298-9c1aea8b27f4" containerName="main" containerID="cri-o://65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4" gracePeriod=45
Nov 03 01:15:58.180706 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.180640    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:15:58.328127 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328091    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-tmp-dir\") pod \"66432f27-b3ec-443d-a298-9c1aea8b27f4\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") "
Nov 03 01:15:58.328330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328225    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66432f27-b3ec-443d-a298-9c1aea8b27f4-tls-certs\" (UniqueName: \"kubernetes.io/secret/66432f27-b3ec-443d-a298-9c1aea8b27f4-tls-certs\") pod \"66432f27-b3ec-443d-a298-9c1aea8b27f4\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") "
Nov 03 01:15:58.328330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328283    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66432f27-b3ec-443d-a298-9c1aea8b27f4-kube-api-access-mqxm6\" (UniqueName: \"kubernetes.io/projected/66432f27-b3ec-443d-a298-9c1aea8b27f4-kube-api-access-mqxm6\") pod \"66432f27-b3ec-443d-a298-9c1aea8b27f4\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") "
Nov 03 01:15:58.328330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328305    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-dshm\") pod \"66432f27-b3ec-443d-a298-9c1aea8b27f4\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") "
Nov 03 01:15:58.328473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328396    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-model-cache\") pod \"66432f27-b3ec-443d-a298-9c1aea8b27f4\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") "
Nov 03 01:15:58.328473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328410    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-tmp-dir" pod "66432f27-b3ec-443d-a298-9c1aea8b27f4" (UID: "66432f27-b3ec-443d-a298-9c1aea8b27f4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:58.328473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328436    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-home\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-home\") pod \"66432f27-b3ec-443d-a298-9c1aea8b27f4\" (UID: \"66432f27-b3ec-443d-a298-9c1aea8b27f4\") "
Nov 03 01:15:58.328658 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328636    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-model-cache" pod "66432f27-b3ec-443d-a298-9c1aea8b27f4" (UID: "66432f27-b3ec-443d-a298-9c1aea8b27f4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:58.328750 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328723    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-home" pod "66432f27-b3ec-443d-a298-9c1aea8b27f4" (UID: "66432f27-b3ec-443d-a298-9c1aea8b27f4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:58.328915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328898    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:58.328983 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328922    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:58.328983 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.328936    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:58.330724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.330703    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-dshm" pod "66432f27-b3ec-443d-a298-9c1aea8b27f4" (UID: "66432f27-b3ec-443d-a298-9c1aea8b27f4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:15:58.330724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.330716    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66432f27-b3ec-443d-a298-9c1aea8b27f4-tls-certs" pod "66432f27-b3ec-443d-a298-9c1aea8b27f4" (UID: "66432f27-b3ec-443d-a298-9c1aea8b27f4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:58.330900 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.330774    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66432f27-b3ec-443d-a298-9c1aea8b27f4-kube-api-access-mqxm6" pod "66432f27-b3ec-443d-a298-9c1aea8b27f4" (UID: "66432f27-b3ec-443d-a298-9c1aea8b27f4"). InnerVolumeSpecName "kube-api-access-mqxm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:58.412060 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.412031    6000 generic.go:356] "Generic (PLEG): container finished" podID="66432f27-b3ec-443d-a298-9c1aea8b27f4" containerID="65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4" exitCode=0
Nov 03 01:15:58.412236 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.412076    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"
Nov 03 01:15:58.412236 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.412078    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx" event={"ID":"66432f27-b3ec-443d-a298-9c1aea8b27f4","Type":"ContainerDied","Data":"65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4"}
Nov 03 01:15:58.412236 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.412187    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx" event={"ID":"66432f27-b3ec-443d-a298-9c1aea8b27f4","Type":"ContainerDied","Data":"813eedcf87676bde0b3eaadf6285dabcab22261a76f3f3b6e9767700694d0b93"}
Nov 03 01:15:58.412236 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.412204    6000 scope.go:122] "RemoveContainer" containerID="65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4"
Nov 03 01:15:58.422142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.422125    6000 scope.go:122] "RemoveContainer" containerID="65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4"
Nov 03 01:15:58.422450 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:15:58.422431    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4\": container with ID starting with 65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4 not found: ID does not exist" containerID="65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4"
Nov 03 01:15:58.422532 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.422457    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4"} err="failed to get container status \"65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4\": rpc error: code = NotFound desc = could not find container \"65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4\": container with ID starting with 65044f51bbcf0049083b6a3178505a693aa3b79ceda717a542f28785c600afa4 not found: ID does not exist"
Nov 03 01:15:58.430301 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.430280    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432f27-b3ec-443d-a298-9c1aea8b27f4-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:58.430384 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.430303    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqxm6\" (UniqueName: \"kubernetes.io/projected/66432f27-b3ec-443d-a298-9c1aea8b27f4-kube-api-access-mqxm6\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:58.430384 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.430312    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/66432f27-b3ec-443d-a298-9c1aea8b27f4-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:58.459948 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.459887    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"]
Nov 03 01:15:58.461546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.461525    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-5c109b/sub-test-model-5c109b-kserve-6dbc85bbd5-24qjx"]
Nov 03 01:15:58.566935 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:58.566904    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66432f27-b3ec-443d-a298-9c1aea8b27f4" path="/var/lib/kubelet/pods/66432f27-b3ec-443d-a298-9c1aea8b27f4/volumes"
Nov 03 01:15:59.249662 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.249629    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"]
Nov 03 01:15:59.249907 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.249857    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb" podUID="0fd57543-8daa-4be4-a9f9-0208bcd55b4e" containerName="maas-api" containerID="cri-o://ece9e01b156f379e5f90508bc7fe84cee2f2ac369a092d9e3eb077fa3d17fe21" gracePeriod=30
Nov 03 01:15:59.419729 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.419697    6000 generic.go:356] "Generic (PLEG): container finished" podID="0fd57543-8daa-4be4-a9f9-0208bcd55b4e" containerID="ece9e01b156f379e5f90508bc7fe84cee2f2ac369a092d9e3eb077fa3d17fe21" exitCode=0
Nov 03 01:15:59.419901 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.419773    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb" event={"ID":"0fd57543-8daa-4be4-a9f9-0208bcd55b4e","Type":"ContainerDied","Data":"ece9e01b156f379e5f90508bc7fe84cee2f2ac369a092d9e3eb077fa3d17fe21"}
Nov 03 01:15:59.511939 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.511859    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 01:15:59.643800 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.643768    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-openshift-service-ca\") pod \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") "
Nov 03 01:15:59.644231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.643888    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-maas-api-tls\") pod \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") "
Nov 03 01:15:59.644231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.643957    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-metrics-certs\") pod \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") "
Nov 03 01:15:59.644231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.644036    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-kube-api-access-fsnzr\" (UniqueName: \"kubernetes.io/projected/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-kube-api-access-fsnzr\") pod \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\" (UID: \"0fd57543-8daa-4be4-a9f9-0208bcd55b4e\") "
Nov 03 01:15:59.644231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.644183    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-openshift-service-ca" pod "0fd57543-8daa-4be4-a9f9-0208bcd55b4e" (UID: "0fd57543-8daa-4be4-a9f9-0208bcd55b4e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:15:59.645084 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.644642    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:59.646339 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.646292    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-maas-api-tls" pod "0fd57543-8daa-4be4-a9f9-0208bcd55b4e" (UID: "0fd57543-8daa-4be4-a9f9-0208bcd55b4e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:59.646479 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.646424    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-kube-api-access-fsnzr" pod "0fd57543-8daa-4be4-a9f9-0208bcd55b4e" (UID: "0fd57543-8daa-4be4-a9f9-0208bcd55b4e"). InnerVolumeSpecName "kube-api-access-fsnzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:15:59.646659 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.646645    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-metrics-certs" pod "0fd57543-8daa-4be4-a9f9-0208bcd55b4e" (UID: "0fd57543-8daa-4be4-a9f9-0208bcd55b4e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:15:59.745273 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.745221    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsnzr\" (UniqueName: \"kubernetes.io/projected/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-kube-api-access-fsnzr\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:59.745273 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.745277    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:15:59.745501 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:15:59.745288    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0fd57543-8daa-4be4-a9f9-0208bcd55b4e-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:00.428309 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:00.428281    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"
Nov 03 01:16:00.428309 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:00.428248    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb" event={"ID":"0fd57543-8daa-4be4-a9f9-0208bcd55b4e","Type":"ContainerDied","Data":"a3f86fb87a8a3f63d1c98e33d936ae57be770fecbf832573603c7e875326b4b1"}
Nov 03 01:16:00.428581 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:00.428369    6000 scope.go:122] "RemoveContainer" containerID="ece9e01b156f379e5f90508bc7fe84cee2f2ac369a092d9e3eb077fa3d17fe21"
Nov 03 01:16:00.477300 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:00.477245    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"]
Nov 03 01:16:00.487958 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:00.487926    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-extmod-a-c738e0-6dc6b76558-mk8mb"]
Nov 03 01:16:00.567759 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:00.567724    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd57543-8daa-4be4-a9f9-0208bcd55b4e" path="/var/lib/kubelet/pods/0fd57543-8daa-4be4-a9f9-0208bcd55b4e/volumes"
Nov 03 01:16:10.066896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:10.066851    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"]
Nov 03 01:16:15.067171 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.067132    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn" podUID="c746ad54-d687-422d-b0f4-3ba0e6f31a55" containerName="payload-processing" containerID="cri-o://d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13" gracePeriod=25
Nov 03 01:16:15.312108 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.312077    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 01:16:15.419248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.419220    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c746ad54-d687-422d-b0f4-3ba0e6f31a55-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c746ad54-d687-422d-b0f4-3ba0e6f31a55-plugins-config-volume\") pod \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") "
Nov 03 01:16:15.419473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.419352    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c746ad54-d687-422d-b0f4-3ba0e6f31a55-kube-api-access-wxt44\" (UniqueName: \"kubernetes.io/projected/c746ad54-d687-422d-b0f4-3ba0e6f31a55-kube-api-access-wxt44\") pod \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\" (UID: \"c746ad54-d687-422d-b0f4-3ba0e6f31a55\") "
Nov 03 01:16:15.419597 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.419572    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c746ad54-d687-422d-b0f4-3ba0e6f31a55-plugins-config-volume" pod "c746ad54-d687-422d-b0f4-3ba0e6f31a55" (UID: "c746ad54-d687-422d-b0f4-3ba0e6f31a55"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:15.419761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.419744    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c746ad54-d687-422d-b0f4-3ba0e6f31a55-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.421605 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.421582    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c746ad54-d687-422d-b0f4-3ba0e6f31a55-kube-api-access-wxt44" pod "c746ad54-d687-422d-b0f4-3ba0e6f31a55" (UID: "c746ad54-d687-422d-b0f4-3ba0e6f31a55"). InnerVolumeSpecName "kube-api-access-wxt44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:15.514178 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.514144    6000 generic.go:356] "Generic (PLEG): container finished" podID="c746ad54-d687-422d-b0f4-3ba0e6f31a55" containerID="d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13" exitCode=0
Nov 03 01:16:15.514351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.514190    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"
Nov 03 01:16:15.514351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.514223    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn" event={"ID":"c746ad54-d687-422d-b0f4-3ba0e6f31a55","Type":"ContainerDied","Data":"d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13"}
Nov 03 01:16:15.514351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.514298    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn" event={"ID":"c746ad54-d687-422d-b0f4-3ba0e6f31a55","Type":"ContainerDied","Data":"af4891b046312f4103344218f163ffd282e3915ecf09c6e1f5fceea6d6bad8f6"}
Nov 03 01:16:15.514351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.514323    6000 scope.go:122] "RemoveContainer" containerID="d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13"
Nov 03 01:16:15.520741 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.520715    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxt44\" (UniqueName: \"kubernetes.io/projected/c746ad54-d687-422d-b0f4-3ba0e6f31a55-kube-api-access-wxt44\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:15.524196 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.524174    6000 scope.go:122] "RemoveContainer" containerID="d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13"
Nov 03 01:16:15.524579 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:15.524547    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13\": container with ID starting with d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13 not found: ID does not exist" containerID="d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13"
Nov 03 01:16:15.524656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.524591    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13"} err="failed to get container status \"d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13\": rpc error: code = NotFound desc = could not find container \"d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13\": container with ID starting with d73833cfac7ac3d9526133417f6b3abd6574b35936493fd0bc6f27d88b64fa13 not found: ID does not exist"
Nov 03 01:16:15.563107 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.563072    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"]
Nov 03 01:16:15.566159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:15.566135    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-0db47e-9695866b9-xxxnn"]
Nov 03 01:16:16.566654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:16.566621    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c746ad54-d687-422d-b0f4-3ba0e6f31a55" path="/var/lib/kubelet/pods/c746ad54-d687-422d-b0f4-3ba0e6f31a55/volumes"
Nov 03 01:16:17.004592 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:17.004561    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:16:18.834345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:18.834309    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"]
Nov 03 01:16:18.867411 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:18.867377    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"]
Nov 03 01:16:19.895052 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:19.895013    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:16:23.835180 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:23.835130    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" podUID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" containerName="payload-processing" containerID="cri-o://44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57" gracePeriod=25
Nov 03 01:16:23.868214 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:23.868177    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" podUID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" containerName="payload-pre-processing" containerID="cri-o://b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0" gracePeriod=25
Nov 03 01:16:28.369054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:28.369016    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" podUID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:28.369054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:28.369033    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" podUID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:30.804619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:30.804582    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"]
Nov 03 01:16:35.805008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:35.804958    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" podUID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" containerName="payload-pre-processing" containerID="cri-o://a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66" gracePeriod=25
Nov 03 01:16:35.911623 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:35.911589    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:16:38.181616 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:38.181578    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"]
Nov 03 01:16:38.215367 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:38.215331    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"]
Nov 03 01:16:38.369394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:38.369355    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" podUID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:38.369623 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:38.369360    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" podUID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:39.687088 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.687055    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"]
Nov 03 01:16:39.687628 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.687279    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r" podUID="619a36e0-3a99-47b1-ae3a-0a9395e118a4" containerName="maas-api" containerID="cri-o://30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706" gracePeriod=30
Nov 03 01:16:39.940031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.939953    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 01:16:39.985927 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.985894    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619a36e0-3a99-47b1-ae3a-0a9395e118a4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/619a36e0-3a99-47b1-ae3a-0a9395e118a4-openshift-service-ca\") pod \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") "
Nov 03 01:16:39.985927 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.985936    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619a36e0-3a99-47b1-ae3a-0a9395e118a4-kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/619a36e0-3a99-47b1-ae3a-0a9395e118a4-kube-api-access-rmjcd\") pod \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") "
Nov 03 01:16:39.986304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.986245    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-maas-api-tls\") pod \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") "
Nov 03 01:16:39.986398 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.986382    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-metrics-certs\") pod \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\" (UID: \"619a36e0-3a99-47b1-ae3a-0a9395e118a4\") "
Nov 03 01:16:39.987075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.987028    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619a36e0-3a99-47b1-ae3a-0a9395e118a4-openshift-service-ca" pod "619a36e0-3a99-47b1-ae3a-0a9395e118a4" (UID: "619a36e0-3a99-47b1-ae3a-0a9395e118a4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:39.988653 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.988629    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-maas-api-tls" pod "619a36e0-3a99-47b1-ae3a-0a9395e118a4" (UID: "619a36e0-3a99-47b1-ae3a-0a9395e118a4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:39.988952 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.988929    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619a36e0-3a99-47b1-ae3a-0a9395e118a4-kube-api-access-rmjcd" pod "619a36e0-3a99-47b1-ae3a-0a9395e118a4" (UID: "619a36e0-3a99-47b1-ae3a-0a9395e118a4"). InnerVolumeSpecName "kube-api-access-rmjcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:39.989363 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:39.989342    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-metrics-certs" pod "619a36e0-3a99-47b1-ae3a-0a9395e118a4" (UID: "619a36e0-3a99-47b1-ae3a-0a9395e118a4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:16:40.087855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.087812    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:40.087855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.087851    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/619a36e0-3a99-47b1-ae3a-0a9395e118a4-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:40.087855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.087862    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/619a36e0-3a99-47b1-ae3a-0a9395e118a4-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:40.088116 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.087871    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmjcd\" (UniqueName: \"kubernetes.io/projected/619a36e0-3a99-47b1-ae3a-0a9395e118a4-kube-api-access-rmjcd\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:40.655969 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.655931    6000 generic.go:356] "Generic (PLEG): container finished" podID="619a36e0-3a99-47b1-ae3a-0a9395e118a4" containerID="30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706" exitCode=0
Nov 03 01:16:40.656170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.655983    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"
Nov 03 01:16:40.656170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.656019    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r" event={"ID":"619a36e0-3a99-47b1-ae3a-0a9395e118a4","Type":"ContainerDied","Data":"30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706"}
Nov 03 01:16:40.656170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.656084    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r" event={"ID":"619a36e0-3a99-47b1-ae3a-0a9395e118a4","Type":"ContainerDied","Data":"c28baeddbdb14503069201529ea138768b388e48f70c71e26cfbe3613fc2e18f"}
Nov 03 01:16:40.656170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.656116    6000 scope.go:122] "RemoveContainer" containerID="30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706"
Nov 03 01:16:40.665986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.665967    6000 scope.go:122] "RemoveContainer" containerID="30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706"
Nov 03 01:16:40.666248 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:40.666227    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706\": container with ID starting with 30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706 not found: ID does not exist" containerID="30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706"
Nov 03 01:16:40.666337 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.666298    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706"} err="failed to get container status \"30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706\": rpc error: code = NotFound desc = could not find container \"30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706\": container with ID starting with 30b3ec21992e1aeaa4878cd9d05c7aee3ba95d1d3591a19ad5f7dacb95215706 not found: ID does not exist"
Nov 03 01:16:40.670384 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.670353    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" podUID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:40.706278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.706231    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"]
Nov 03 01:16:40.709340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:40.709318    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-20109d-6d74955bb5-mpl6r"]
Nov 03 01:16:42.566394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:42.566309    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619a36e0-3a99-47b1-ae3a-0a9395e118a4" path="/var/lib/kubelet/pods/619a36e0-3a99-47b1-ae3a-0a9395e118a4/volumes"
Nov 03 01:16:43.182781 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:43.182735    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" podUID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" containerName="payload-processing" containerID="cri-o://cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37" gracePeriod=25
Nov 03 01:16:43.216600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:43.216540    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" podUID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" containerName="payload-pre-processing" containerID="cri-o://18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914" gracePeriod=25
Nov 03 01:16:44.139341 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.139311    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 01:16:44.162317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.162293    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 01:16:44.233158 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233121    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de6eba37-c7cd-4239-a7fd-9b815b14dc79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6eba37-c7cd-4239-a7fd-9b815b14dc79-plugins-config-volume\") pod \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") "
Nov 03 01:16:44.233379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233182    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-plugins-config-volume\") pod \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") "
Nov 03 01:16:44.233379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233373    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-kube-api-access-rzssj\" (UniqueName: \"kubernetes.io/projected/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-kube-api-access-rzssj\") pod \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\" (UID: \"68f9048e-4dfe-4376-90e6-58c49ff9fc8c\") "
Nov 03 01:16:44.233514 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233421    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de6eba37-c7cd-4239-a7fd-9b815b14dc79-kube-api-access-n874q\" (UniqueName: \"kubernetes.io/projected/de6eba37-c7cd-4239-a7fd-9b815b14dc79-kube-api-access-n874q\") pod \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\" (UID: \"de6eba37-c7cd-4239-a7fd-9b815b14dc79\") "
Nov 03 01:16:44.233657 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233630    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de6eba37-c7cd-4239-a7fd-9b815b14dc79-plugins-config-volume" pod "de6eba37-c7cd-4239-a7fd-9b815b14dc79" (UID: "de6eba37-c7cd-4239-a7fd-9b815b14dc79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:44.233657 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233643    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-plugins-config-volume" pod "68f9048e-4dfe-4376-90e6-58c49ff9fc8c" (UID: "68f9048e-4dfe-4376-90e6-58c49ff9fc8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:44.233876 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233856    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/de6eba37-c7cd-4239-a7fd-9b815b14dc79-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:44.233941 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.233886    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:44.235869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.235846    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-kube-api-access-rzssj" pod "68f9048e-4dfe-4376-90e6-58c49ff9fc8c" (UID: "68f9048e-4dfe-4376-90e6-58c49ff9fc8c"). InnerVolumeSpecName "kube-api-access-rzssj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:44.236212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.236184    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de6eba37-c7cd-4239-a7fd-9b815b14dc79-kube-api-access-n874q" pod "de6eba37-c7cd-4239-a7fd-9b815b14dc79" (UID: "de6eba37-c7cd-4239-a7fd-9b815b14dc79"). InnerVolumeSpecName "kube-api-access-n874q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:44.334816 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.334785    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rzssj\" (UniqueName: \"kubernetes.io/projected/68f9048e-4dfe-4376-90e6-58c49ff9fc8c-kube-api-access-rzssj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:44.334816 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.334813    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n874q\" (UniqueName: \"kubernetes.io/projected/de6eba37-c7cd-4239-a7fd-9b815b14dc79-kube-api-access-n874q\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:44.680272 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.680228    6000 generic.go:356] "Generic (PLEG): container finished" podID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" containerID="b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0" exitCode=0
Nov 03 01:16:44.680478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.680304    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"
Nov 03 01:16:44.680478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.680303    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" event={"ID":"de6eba37-c7cd-4239-a7fd-9b815b14dc79","Type":"ContainerDied","Data":"b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0"}
Nov 03 01:16:44.680478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.680437    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx" event={"ID":"de6eba37-c7cd-4239-a7fd-9b815b14dc79","Type":"ContainerDied","Data":"98200b5dbee90990284130f49a5b84a7a7f25c121c15fe76999deaafcefe4d3e"}
Nov 03 01:16:44.680478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.680469    6000 scope.go:122] "RemoveContainer" containerID="b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0"
Nov 03 01:16:44.684505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.683851    6000 generic.go:356] "Generic (PLEG): container finished" podID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" containerID="44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57" exitCode=0
Nov 03 01:16:44.684505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.683951    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" event={"ID":"68f9048e-4dfe-4376-90e6-58c49ff9fc8c","Type":"ContainerDied","Data":"44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57"}
Nov 03 01:16:44.684505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.683989    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd" event={"ID":"68f9048e-4dfe-4376-90e6-58c49ff9fc8c","Type":"ContainerDied","Data":"968ccd6ff6399613abd96143f898262e06431c7e7da705e9458f6a51671a3d2a"}
Nov 03 01:16:44.684505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.684066    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"
Nov 03 01:16:44.692758 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.692739    6000 scope.go:122] "RemoveContainer" containerID="b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0"
Nov 03 01:16:44.693045 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:44.693018    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0\": container with ID starting with b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0 not found: ID does not exist" containerID="b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0"
Nov 03 01:16:44.693137 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.693059    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0"} err="failed to get container status \"b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0\": rpc error: code = NotFound desc = could not find container \"b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0\": container with ID starting with b3ba5cce3f9dd8eefd24839043d9ba32c46ec9f246e0eab4634acf97917427a0 not found: ID does not exist"
Nov 03 01:16:44.693137 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.693086    6000 scope.go:122] "RemoveContainer" containerID="44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57"
Nov 03 01:16:44.701433 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.701416    6000 scope.go:122] "RemoveContainer" containerID="44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57"
Nov 03 01:16:44.701676 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:44.701660    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57\": container with ID starting with 44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57 not found: ID does not exist" containerID="44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57"
Nov 03 01:16:44.701725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.701684    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57"} err="failed to get container status \"44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57\": rpc error: code = NotFound desc = could not find container \"44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57\": container with ID starting with 44601f54c6da456263aae79fe6e2c19323ca5206f145521db7a8ef769682fd57 not found: ID does not exist"
Nov 03 01:16:44.727758 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.727725    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"]
Nov 03 01:16:44.739052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.739025    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-9908e8-6f794966f5-69xbx"]
Nov 03 01:16:44.762673 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.762641    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"]
Nov 03 01:16:44.766995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.766969    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-9908e8-955f5c679-2r5sd"]
Nov 03 01:16:44.958492 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.958450    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"]
Nov 03 01:16:44.958730 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:44.958701    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy" containerID="cri-o://9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133" gracePeriod=30
Nov 03 01:16:46.566208 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:46.566173    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" path="/var/lib/kubelet/pods/68f9048e-4dfe-4376-90e6-58c49ff9fc8c/volumes"
Nov 03 01:16:46.566639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:46.566597    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" path="/var/lib/kubelet/pods/de6eba37-c7cd-4239-a7fd-9b815b14dc79/volumes"
Nov 03 01:16:47.389469 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:47.389428    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"]
Nov 03 01:16:47.389704 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:47.389677    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy" containerID="cri-o://7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116" gracePeriod=30
Nov 03 01:16:48.739429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:48.739386    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" podUID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:49.746425 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:49.746389    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" podUID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:49.895794 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:49.895755    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:16:50.211653 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.211629    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:16:50.298556 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298526    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-token\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-token\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.298754 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298583    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-kube-api-access-jqbv2\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-kube-api-access-jqbv2\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.298754 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298628    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-podinfo\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.298754 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298652    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-credential-socket\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.298754 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298669    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-socket\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.298754 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298735    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-envoy\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.299069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298787    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-data\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.299069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298814    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-certs\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.299069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.298841    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istiod-ca-cert\") pod \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\" (UID: \"bd96e482-f9fd-4acb-b4eb-195f6c98d526\") "
Nov 03 01:16:50.299069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299048    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-socket" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.299338 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299131    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-data" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.299338 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299322    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-certs" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.299457 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299219    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-credential-socket" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.299523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299460    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.299523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299482    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.299523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299497    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.299523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299500    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istiod-ca-cert" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:50.299523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.299512    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.301314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.301290    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-token" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:50.301710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.301582    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-kube-api-access-jqbv2" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "kube-api-access-jqbv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:50.301710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.301658    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-podinfo" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:50.301710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.301677    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-envoy" pod "bd96e482-f9fd-4acb-b4eb-195f6c98d526" (UID: "bd96e482-f9fd-4acb-b4eb-195f6c98d526"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:50.400923 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.400892    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.400923 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.400919    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqbv2\" (UniqueName: \"kubernetes.io/projected/bd96e482-f9fd-4acb-b4eb-195f6c98d526-kube-api-access-jqbv2\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.401135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.400931    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.401135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.400939    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.401135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.400948    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd96e482-f9fd-4acb-b4eb-195f6c98d526-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:50.670292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.670170    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" podUID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:50.718945 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.718912    6000 generic.go:356] "Generic (PLEG): container finished" podID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerID="9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133" exitCode=0
Nov 03 01:16:50.719110 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.718962    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"
Nov 03 01:16:50.719110 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.718998    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" event={"ID":"bd96e482-f9fd-4acb-b4eb-195f6c98d526","Type":"ContainerDied","Data":"9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133"}
Nov 03 01:16:50.719110 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.719052    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv" event={"ID":"bd96e482-f9fd-4acb-b4eb-195f6c98d526","Type":"ContainerDied","Data":"68cccde5936a1e2eab63b1cf9679cadd35eb81c9202634b18379416b39e0c403"}
Nov 03 01:16:50.719110 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.719075    6000 scope.go:122] "RemoveContainer" containerID="9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133"
Nov 03 01:16:50.728547 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.728527    6000 scope.go:122] "RemoveContainer" containerID="9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133"
Nov 03 01:16:50.728786 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:50.728766    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133\": container with ID starting with 9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133 not found: ID does not exist" containerID="9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133"
Nov 03 01:16:50.728840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.728793    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133"} err="failed to get container status \"9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133\": rpc error: code = NotFound desc = could not find container \"9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133\": container with ID starting with 9ad32c5de1a17114dced70f9c104ef964b480c3cad96d3483c95cd987fcfb133 not found: ID does not exist"
Nov 03 01:16:50.767048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.767019    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"]
Nov 03 01:16:50.794986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:50.794949    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-099583-openshift-default-59874d54f-w6tqv"]
Nov 03 01:16:51.358694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:51.358661    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:16:51.501285 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:51.501224    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"]
Nov 03 01:16:51.552725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:51.552688    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"]
Nov 03 01:16:52.570771 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.570739    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" path="/var/lib/kubelet/pods/bd96e482-f9fd-4acb-b4eb-195f6c98d526/volumes"
Nov 03 01:16:52.659185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.659118    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:16:52.728643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728604    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istiod-ca-cert\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.728825 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728681    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-data\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.728825 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728730    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-socket\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.728825 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728776    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-envoy\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.728825 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728818    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-kube-api-access-lr6sm\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-kube-api-access-lr6sm\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.729076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728853    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-token\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-token\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.729076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728926    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-podinfo\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.729076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.728987    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-certs\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.729076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.729041    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-data" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.729076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.729067    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istiod-ca-cert" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:52.729388 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.729081    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-credential-socket\") pod \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\" (UID: \"09ab309f-7e1a-4789-b54e-f609b6c94ad6\") "
Nov 03 01:16:52.729388 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.729299    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-credential-socket" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.729933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.729910    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.730110 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.730095    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.730298 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.730270    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-socket" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.730630 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.730291    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.730851 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.730826    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-certs" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.731971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.731945    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-token" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:52.732088 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.731979    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-kube-api-access-lr6sm" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "kube-api-access-lr6sm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:52.732945 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.732917    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-envoy" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:16:52.733711 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.733686    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-podinfo" pod "09ab309f-7e1a-4789-b54e-f609b6c94ad6" (UID: "09ab309f-7e1a-4789-b54e-f609b6c94ad6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:16:52.739005 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.738978    6000 generic.go:356] "Generic (PLEG): container finished" podID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerID="7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116" exitCode=0
Nov 03 01:16:52.739131 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.739036    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"
Nov 03 01:16:52.739131 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.739063    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" event={"ID":"09ab309f-7e1a-4789-b54e-f609b6c94ad6","Type":"ContainerDied","Data":"7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116"}
Nov 03 01:16:52.739331 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.739156    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986" event={"ID":"09ab309f-7e1a-4789-b54e-f609b6c94ad6","Type":"ContainerDied","Data":"740332da218db14f67a77bab3ff9e2d8e6082c13bcb391b4914c8aac0cf813f4"}
Nov 03 01:16:52.739331 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.739188    6000 scope.go:122] "RemoveContainer" containerID="7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116"
Nov 03 01:16:52.756144 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.756121    6000 scope.go:122] "RemoveContainer" containerID="7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116"
Nov 03 01:16:52.756489 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:52.756466    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116\": container with ID starting with 7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116 not found: ID does not exist" containerID="7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116"
Nov 03 01:16:52.756579 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.756500    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116"} err="failed to get container status \"7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116\": rpc error: code = NotFound desc = could not find container \"7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116\": container with ID starting with 7ecf41d7001f01954d32af8404ce32660ab16d23765a59fe855d448e22afc116 not found: ID does not exist"
Nov 03 01:16:52.788585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.788548    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"]
Nov 03 01:16:52.791955 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.791934    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-20109d-openshift-default-fdcf648d5-hn986"]
Nov 03 01:16:52.832016 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.831992    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.832016 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.832014    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.832185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.832025    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr6sm\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-kube-api-access-lr6sm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.832185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.832036    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.832185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.832045    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09ab309f-7e1a-4789-b54e-f609b6c94ad6-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:52.832185 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:52.832053    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09ab309f-7e1a-4789-b54e-f609b6c94ad6-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:54.565778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:54.565742    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" path="/var/lib/kubelet/pods/09ab309f-7e1a-4789-b54e-f609b6c94ad6/volumes"
Nov 03 01:16:56.059690 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.059667    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 01:16:56.166227 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.166196    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-kube-api-access-cc4g9\" (UniqueName: \"kubernetes.io/projected/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-kube-api-access-cc4g9\") pod \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") "
Nov 03 01:16:56.166421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.166330    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-plugins-config-volume\") pod \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\" (UID: \"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f\") "
Nov 03 01:16:56.166743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.166716    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-plugins-config-volume" pod "b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" (UID: "b0f400e1-4d6e-41e3-8dd9-7368268d6a4f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:16:56.168301 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.168278    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-kube-api-access-cc4g9" pod "b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" (UID: "b0f400e1-4d6e-41e3-8dd9-7368268d6a4f"). InnerVolumeSpecName "kube-api-access-cc4g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:16:56.268010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.267974    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc4g9\" (UniqueName: \"kubernetes.io/projected/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-kube-api-access-cc4g9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:56.268010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.268005    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:16:56.502427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.502329    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" podUID="1e39d791-541f-4cc3-baec-5635b01020a6" containerName="payload-processing" containerID="cri-o://8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd" gracePeriod=25
Nov 03 01:16:56.553554 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.553512    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" podUID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" containerName="payload-pre-processing" containerID="cri-o://0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7" gracePeriod=25
Nov 03 01:16:56.763362 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.763275    6000 generic.go:356] "Generic (PLEG): container finished" podID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" containerID="a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66" exitCode=0
Nov 03 01:16:56.763362 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.763325    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"
Nov 03 01:16:56.763362 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.763344    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" event={"ID":"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f","Type":"ContainerDied","Data":"a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66"}
Nov 03 01:16:56.763622 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.763406    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt" event={"ID":"b0f400e1-4d6e-41e3-8dd9-7368268d6a4f","Type":"ContainerDied","Data":"a62680d345b8853bdde7a828cc351341d453c85da35970d12705c9f41c6d2b91"}
Nov 03 01:16:56.763622 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.763437    6000 scope.go:122] "RemoveContainer" containerID="a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66"
Nov 03 01:16:56.772782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.772763    6000 scope.go:122] "RemoveContainer" containerID="a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66"
Nov 03 01:16:56.773060 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:16:56.773041    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66\": container with ID starting with a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66 not found: ID does not exist" containerID="a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66"
Nov 03 01:16:56.773118 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.773070    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66"} err="failed to get container status \"a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66\": rpc error: code = NotFound desc = could not find container \"a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66\": container with ID starting with a192bbe32e81d08e92c4b19fecbdffdfb3889cff67985cf7fe23918609a71d66 not found: ID does not exist"
Nov 03 01:16:56.834962 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.834938    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"]
Nov 03 01:16:56.853954 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:56.853930    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-099583-568499c68c-nf6zt"]
Nov 03 01:16:58.572204 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:58.572164    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" path="/var/lib/kubelet/pods/b0f400e1-4d6e-41e3-8dd9-7368268d6a4f/volumes"
Nov 03 01:16:58.739898 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:58.739863    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" podUID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:59.040289 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:59.040216    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"]
Nov 03 01:16:59.041440 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:59.040472    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy" containerID="cri-o://f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648" gracePeriod=30
Nov 03 01:16:59.745708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:59.745671    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" podUID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:16:59.825631 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:59.825595    6000 patch_prober.go:29] interesting pod/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 03 01:16:59.825799 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:16:59.825657    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 03 01:17:00.345636 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:00.345600    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" podUID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:00.345636 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:00.345620    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" podUID="1e39d791-541f-4cc3-baec-5635b01020a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:03.472194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.472167    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 01:17:03.512845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.512813    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 01:17:03.643622 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.643591    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-kube-api-access-bs6hz\" (UniqueName: \"kubernetes.io/projected/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-kube-api-access-bs6hz\") pod \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") "
Nov 03 01:17:03.643801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.643658    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-plugins-config-volume\") pod \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") "
Nov 03 01:17:03.643801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.643771    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-plugins-config-volume\") pod \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\" (UID: \"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6\") "
Nov 03 01:17:03.643954 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.643893    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-kube-api-access-2n9gm\" (UniqueName: \"kubernetes.io/projected/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-kube-api-access-2n9gm\") pod \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\" (UID: \"3adb0154-70eb-464b-bfcb-3ffc3d5476e0\") "
Nov 03 01:17:03.644069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.644050    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-plugins-config-volume" pod "3adb0154-70eb-464b-bfcb-3ffc3d5476e0" (UID: "3adb0154-70eb-464b-bfcb-3ffc3d5476e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:03.644169 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.644148    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-plugins-config-volume" pod "e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" (UID: "e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:03.644412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.644393    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.644489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.644417    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.645892 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.645872    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-kube-api-access-bs6hz" pod "e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" (UID: "e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6"). InnerVolumeSpecName "kube-api-access-bs6hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:03.646086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.646067    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-kube-api-access-2n9gm" pod "3adb0154-70eb-464b-bfcb-3ffc3d5476e0" (UID: "3adb0154-70eb-464b-bfcb-3ffc3d5476e0"). InnerVolumeSpecName "kube-api-access-2n9gm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:03.745403 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.745310    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2n9gm\" (UniqueName: \"kubernetes.io/projected/3adb0154-70eb-464b-bfcb-3ffc3d5476e0-kube-api-access-2n9gm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.745403 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.745339    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs6hz\" (UniqueName: \"kubernetes.io/projected/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6-kube-api-access-bs6hz\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:03.804459 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.804421    6000 generic.go:356] "Generic (PLEG): container finished" podID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" containerID="18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914" exitCode=0
Nov 03 01:17:03.804664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.804471    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"
Nov 03 01:17:03.804664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.804501    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" event={"ID":"3adb0154-70eb-464b-bfcb-3ffc3d5476e0","Type":"ContainerDied","Data":"18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914"}
Nov 03 01:17:03.804664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.804563    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm" event={"ID":"3adb0154-70eb-464b-bfcb-3ffc3d5476e0","Type":"ContainerDied","Data":"f684623fbfe6a64bda4f4763d8a4cb440ab415bd285033d2a66dfe8c37964904"}
Nov 03 01:17:03.804664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.804599    6000 scope.go:122] "RemoveContainer" containerID="18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914"
Nov 03 01:17:03.805937 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.805845    6000 generic.go:356] "Generic (PLEG): container finished" podID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" containerID="cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37" exitCode=0
Nov 03 01:17:03.805937 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.805898    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" event={"ID":"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6","Type":"ContainerDied","Data":"cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37"}
Nov 03 01:17:03.805937 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.805907    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"
Nov 03 01:17:03.806165 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.805938    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f" event={"ID":"e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6","Type":"ContainerDied","Data":"8f35ee88c56e8c8bd01a801889fda45cb175c82d9a2bde717a8f17c313727c72"}
Nov 03 01:17:03.814128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.814111    6000 scope.go:122] "RemoveContainer" containerID="18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914"
Nov 03 01:17:03.814457 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:03.814433    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914\": container with ID starting with 18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914 not found: ID does not exist" containerID="18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914"
Nov 03 01:17:03.814529 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.814470    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914"} err="failed to get container status \"18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914\": rpc error: code = NotFound desc = could not find container \"18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914\": container with ID starting with 18d12d18685796402b2aa28d9f920e77411ab8ccbe0946c1e7b3ed8b1b758914 not found: ID does not exist"
Nov 03 01:17:03.814529 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.814495    6000 scope.go:122] "RemoveContainer" containerID="cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37"
Nov 03 01:17:03.823527 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.823509    6000 scope.go:122] "RemoveContainer" containerID="cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37"
Nov 03 01:17:03.823803 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:03.823786    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37\": container with ID starting with cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37 not found: ID does not exist" containerID="cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37"
Nov 03 01:17:03.823853 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.823812    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37"} err="failed to get container status \"cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37\": rpc error: code = NotFound desc = could not find container \"cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37\": container with ID starting with cb65ef4b6ef2c88f1df4d528c98033d38c76b01819e5ecabb381c77637f72c37 not found: ID does not exist"
Nov 03 01:17:03.851000 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.850968    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"]
Nov 03 01:17:03.853077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.853054    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-20109d-6955788f79-7t47f"]
Nov 03 01:17:03.884231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.884195    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"]
Nov 03 01:17:03.885946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:03.885921    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-20109d-699d485b79-c87cm"]
Nov 03 01:17:04.148052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.148010    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:17:04.307361 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.307339    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 01:17:04.453659 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453624    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-kube-api-access-rx9vs\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-kube-api-access-rx9vs\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.453819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453674    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-socket\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.453819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453710    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-token\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-token\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.453819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453738    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-credential-socket\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.453819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453785    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-envoy\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.454062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453873    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14b40ec4-3b82-4726-a95b-f80577c4b051-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b40ec4-3b82-4726-a95b-f80577c4b051-istiod-ca-cert\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.454062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453899    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-certs\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.454062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.453937    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-podinfo\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.454062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454016    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-data\") pod \"14b40ec4-3b82-4726-a95b-f80577c4b051\" (UID: \"14b40ec4-3b82-4726-a95b-f80577c4b051\") "
Nov 03 01:17:04.454346 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454057    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-credential-socket" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.454346 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454069    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-socket" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.454346 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454304    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-certs" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.454346 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454336    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14b40ec4-3b82-4726-a95b-f80577c4b051-istiod-ca-cert" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:04.454705 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454578    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.454705 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454613    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.454705 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454628    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/14b40ec4-3b82-4726-a95b-f80577c4b051-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.454705 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454639    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.454705 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.454676    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-data" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.456171 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.456149    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-token" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:04.456313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.456202    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-envoy" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:04.456602 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.456586    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-kube-api-access-rx9vs" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "kube-api-access-rx9vs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:04.456654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.456612    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-podinfo" pod "14b40ec4-3b82-4726-a95b-f80577c4b051" (UID: "14b40ec4-3b82-4726-a95b-f80577c4b051"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:04.532446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.532416    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"]
Nov 03 01:17:04.540697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.540662    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"]
Nov 03 01:17:04.555656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.555626    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.555656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.555651    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.555833 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.555664    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx9vs\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-kube-api-access-rx9vs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.555833 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.555676    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.555833 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.555690    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/14b40ec4-3b82-4726-a95b-f80577c4b051-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:04.565523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.565498    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" path="/var/lib/kubelet/pods/3adb0154-70eb-464b-bfcb-3ffc3d5476e0/volumes"
Nov 03 01:17:04.566047 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.566028    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" path="/var/lib/kubelet/pods/e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6/volumes"
Nov 03 01:17:04.811978 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.811889    6000 generic.go:356] "Generic (PLEG): container finished" podID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerID="f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648" exitCode=0
Nov 03 01:17:04.811978 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.811939    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"
Nov 03 01:17:04.812247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.811972    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" event={"ID":"14b40ec4-3b82-4726-a95b-f80577c4b051","Type":"ContainerDied","Data":"f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648"}
Nov 03 01:17:04.812247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.812022    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km" event={"ID":"14b40ec4-3b82-4726-a95b-f80577c4b051","Type":"ContainerDied","Data":"37b21c80b0d69f63725bf9144738bbb4621dc2dbf7e29076a26075b11ff7f1e2"}
Nov 03 01:17:04.812247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.812049    6000 scope.go:122] "RemoveContainer" containerID="f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648"
Nov 03 01:17:04.821376 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.821361    6000 scope.go:122] "RemoveContainer" containerID="61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6"
Nov 03 01:17:04.829956 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.829940    6000 scope.go:122] "RemoveContainer" containerID="f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648"
Nov 03 01:17:04.830224 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:04.830204    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648\": container with ID starting with f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648 not found: ID does not exist" containerID="f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648"
Nov 03 01:17:04.830307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.830233    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648"} err="failed to get container status \"f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648\": rpc error: code = NotFound desc = could not find container \"f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648\": container with ID starting with f3fea893bfa36c2a15c6026fbcb34abfff7ab2d4aca5e5dc216c0e02756bf648 not found: ID does not exist"
Nov 03 01:17:04.830307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.830249    6000 scope.go:122] "RemoveContainer" containerID="61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6"
Nov 03 01:17:04.830497 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:04.830481    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6\": container with ID starting with 61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6 not found: ID does not exist" containerID="61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6"
Nov 03 01:17:04.830549 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.830502    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6"} err="failed to get container status \"61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6\": rpc error: code = NotFound desc = could not find container \"61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6\": container with ID starting with 61ce50424f78f85cd5bfa8ace697d0a78ee67fade3abb5313b15c4659cabb4c6 not found: ID does not exist"
Nov 03 01:17:04.859472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.859444    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"]
Nov 03 01:17:04.869509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:04.869488    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-5c109b-openshift-default-7b95fb8859-wv8km"]
Nov 03 01:17:06.565760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:06.565709    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" path="/var/lib/kubelet/pods/14b40ec4-3b82-4726-a95b-f80577c4b051/volumes"
Nov 03 01:17:10.345547 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:10.345506    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" podUID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:10.346053 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:10.345520    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" podUID="1e39d791-541f-4cc3-baec-5635b01020a6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:17:12.115807 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.115768    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"]
Nov 03 01:17:12.116280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.116045    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n" podUID="de14e2d3-979f-4186-91fe-6f8630291d8a" containerName="maas-api" containerID="cri-o://acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f" gracePeriod=30
Nov 03 01:17:12.373460 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.373397    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 01:17:12.442883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.442846    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-metrics-certs\") pod \"de14e2d3-979f-4186-91fe-6f8630291d8a\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") "
Nov 03 01:17:12.443075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.442896    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de14e2d3-979f-4186-91fe-6f8630291d8a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de14e2d3-979f-4186-91fe-6f8630291d8a-openshift-service-ca\") pod \"de14e2d3-979f-4186-91fe-6f8630291d8a\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") "
Nov 03 01:17:12.443075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.442967    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls\") pod \"de14e2d3-979f-4186-91fe-6f8630291d8a\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") "
Nov 03 01:17:12.443075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.442995    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de14e2d3-979f-4186-91fe-6f8630291d8a-kube-api-access-nqcnw\" (UniqueName: \"kubernetes.io/projected/de14e2d3-979f-4186-91fe-6f8630291d8a-kube-api-access-nqcnw\") pod \"de14e2d3-979f-4186-91fe-6f8630291d8a\" (UID: \"de14e2d3-979f-4186-91fe-6f8630291d8a\") "
Nov 03 01:17:12.443417 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.443388    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de14e2d3-979f-4186-91fe-6f8630291d8a-openshift-service-ca" pod "de14e2d3-979f-4186-91fe-6f8630291d8a" (UID: "de14e2d3-979f-4186-91fe-6f8630291d8a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:12.445378 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.445353    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-metrics-certs" pod "de14e2d3-979f-4186-91fe-6f8630291d8a" (UID: "de14e2d3-979f-4186-91fe-6f8630291d8a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:12.445488 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.445406    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de14e2d3-979f-4186-91fe-6f8630291d8a-kube-api-access-nqcnw" pod "de14e2d3-979f-4186-91fe-6f8630291d8a" (UID: "de14e2d3-979f-4186-91fe-6f8630291d8a"). InnerVolumeSpecName "kube-api-access-nqcnw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:12.445555 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.445500    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls" pod "de14e2d3-979f-4186-91fe-6f8630291d8a" (UID: "de14e2d3-979f-4186-91fe-6f8630291d8a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:12.544703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.544669    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:12.544703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.544698    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de14e2d3-979f-4186-91fe-6f8630291d8a-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:12.544703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.544708    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de14e2d3-979f-4186-91fe-6f8630291d8a-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:12.544947 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.544716    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqcnw\" (UniqueName: \"kubernetes.io/projected/de14e2d3-979f-4186-91fe-6f8630291d8a-kube-api-access-nqcnw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:12.857856 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.857823    6000 generic.go:356] "Generic (PLEG): container finished" podID="de14e2d3-979f-4186-91fe-6f8630291d8a" containerID="acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f" exitCode=0
Nov 03 01:17:12.858034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.857874    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"
Nov 03 01:17:12.858034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.857910    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n" event={"ID":"de14e2d3-979f-4186-91fe-6f8630291d8a","Type":"ContainerDied","Data":"acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f"}
Nov 03 01:17:12.858034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.857968    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n" event={"ID":"de14e2d3-979f-4186-91fe-6f8630291d8a","Type":"ContainerDied","Data":"9e3dfc7d7ddcf5a093c232eca4850da3bf4036fdbf5fc9efa9ca8c79f3d73308"}
Nov 03 01:17:12.858034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.857994    6000 scope.go:122] "RemoveContainer" containerID="acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f"
Nov 03 01:17:12.867159 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.867137    6000 scope.go:122] "RemoveContainer" containerID="acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f"
Nov 03 01:17:12.867505 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:12.867483    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f\": container with ID starting with acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f not found: ID does not exist" containerID="acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f"
Nov 03 01:17:12.867599 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.867513    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f"} err="failed to get container status \"acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f\": rpc error: code = NotFound desc = could not find container \"acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f\": container with ID starting with acb1e585db9c661e9cbe68188c27719a7651561e790de06803a20c38003f893f not found: ID does not exist"
Nov 03 01:17:12.916767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.916735    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"]
Nov 03 01:17:12.928129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:12.928086    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-99bdb8-648cf66c85-2l22n"]
Nov 03 01:17:14.566824 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:14.566787    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de14e2d3-979f-4186-91fe-6f8630291d8a" path="/var/lib/kubelet/pods/de14e2d3-979f-4186-91fe-6f8630291d8a/volumes"
Nov 03 01:17:16.671063 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.671023    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:17:16.833016 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.832989    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 01:17:16.835780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.835758    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 01:17:16.882491 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.882456    6000 generic.go:356] "Generic (PLEG): container finished" podID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" containerID="0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7" exitCode=0
Nov 03 01:17:16.882672 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.882517    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"
Nov 03 01:17:16.882672 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.882544    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" event={"ID":"caac0b46-9a05-41c3-8c6a-cf9097d5cf37","Type":"ContainerDied","Data":"0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7"}
Nov 03 01:17:16.882672 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.882620    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt" event={"ID":"caac0b46-9a05-41c3-8c6a-cf9097d5cf37","Type":"ContainerDied","Data":"6e0b7940a4ac7bc47e14c1137ce877a755422144732945e740e29c6dbc9f990f"}
Nov 03 01:17:16.882672 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.882652    6000 scope.go:122] "RemoveContainer" containerID="0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7"
Nov 03 01:17:16.884077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.884051    6000 generic.go:356] "Generic (PLEG): container finished" podID="1e39d791-541f-4cc3-baec-5635b01020a6" containerID="8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd" exitCode=0
Nov 03 01:17:16.884202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.884094    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"
Nov 03 01:17:16.884296 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.884214    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" event={"ID":"1e39d791-541f-4cc3-baec-5635b01020a6","Type":"ContainerDied","Data":"8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd"}
Nov 03 01:17:16.884296 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.884289    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g" event={"ID":"1e39d791-541f-4cc3-baec-5635b01020a6","Type":"ContainerDied","Data":"af931b761454c336eaf649c43970a50bcb560137900191c5bb7493f1a407dc18"}
Nov 03 01:17:16.886893 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.886859    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e39d791-541f-4cc3-baec-5635b01020a6-kube-api-access-md2sw\" (UniqueName: \"kubernetes.io/projected/1e39d791-541f-4cc3-baec-5635b01020a6-kube-api-access-md2sw\") pod \"1e39d791-541f-4cc3-baec-5635b01020a6\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") "
Nov 03 01:17:16.886994 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.886927    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e39d791-541f-4cc3-baec-5635b01020a6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e39d791-541f-4cc3-baec-5635b01020a6-plugins-config-volume\") pod \"1e39d791-541f-4cc3-baec-5635b01020a6\" (UID: \"1e39d791-541f-4cc3-baec-5635b01020a6\") "
Nov 03 01:17:16.887059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.886991    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-kube-api-access-jmn2r\" (UniqueName: \"kubernetes.io/projected/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-kube-api-access-jmn2r\") pod \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") "
Nov 03 01:17:16.887788 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.887123    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-plugins-config-volume\") pod \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\" (UID: \"caac0b46-9a05-41c3-8c6a-cf9097d5cf37\") "
Nov 03 01:17:16.887788 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.887484    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e39d791-541f-4cc3-baec-5635b01020a6-plugins-config-volume" pod "1e39d791-541f-4cc3-baec-5635b01020a6" (UID: "1e39d791-541f-4cc3-baec-5635b01020a6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:16.887788 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.887746    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-plugins-config-volume" pod "caac0b46-9a05-41c3-8c6a-cf9097d5cf37" (UID: "caac0b46-9a05-41c3-8c6a-cf9097d5cf37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:16.888621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.888592    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:16.888733 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.888625    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e39d791-541f-4cc3-baec-5635b01020a6-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:16.889439 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.889415    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e39d791-541f-4cc3-baec-5635b01020a6-kube-api-access-md2sw" pod "1e39d791-541f-4cc3-baec-5635b01020a6" (UID: "1e39d791-541f-4cc3-baec-5635b01020a6"). InnerVolumeSpecName "kube-api-access-md2sw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:16.889547 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.889518    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-kube-api-access-jmn2r" pod "caac0b46-9a05-41c3-8c6a-cf9097d5cf37" (UID: "caac0b46-9a05-41c3-8c6a-cf9097d5cf37"). InnerVolumeSpecName "kube-api-access-jmn2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:16.894812 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.894788    6000 scope.go:122] "RemoveContainer" containerID="0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7"
Nov 03 01:17:16.895114 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:16.895094    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7\": container with ID starting with 0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7 not found: ID does not exist" containerID="0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7"
Nov 03 01:17:16.895212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.895128    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7"} err="failed to get container status \"0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7\": rpc error: code = NotFound desc = could not find container \"0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7\": container with ID starting with 0c653d2b6defdcc8ea9b43faef023fe417efbaf942250b08718f6663c3cf2bc7 not found: ID does not exist"
Nov 03 01:17:16.895212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.895152    6000 scope.go:122] "RemoveContainer" containerID="8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd"
Nov 03 01:17:16.904031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.904004    6000 scope.go:122] "RemoveContainer" containerID="8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd"
Nov 03 01:17:16.904310 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:16.904291    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd\": container with ID starting with 8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd not found: ID does not exist" containerID="8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd"
Nov 03 01:17:16.904373 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.904320    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd"} err="failed to get container status \"8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd\": rpc error: code = NotFound desc = could not find container \"8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd\": container with ID starting with 8c3f495302e5d8a0b707bcb706b74688dc69de9b8c5aefd3b16d7ee0142530cd not found: ID does not exist"
Nov 03 01:17:16.989675 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.989642    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md2sw\" (UniqueName: \"kubernetes.io/projected/1e39d791-541f-4cc3-baec-5635b01020a6-kube-api-access-md2sw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:16.989675 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:16.989668    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmn2r\" (UniqueName: \"kubernetes.io/projected/caac0b46-9a05-41c3-8c6a-cf9097d5cf37-kube-api-access-jmn2r\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:17.233799 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:17.233766    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"]
Nov 03 01:17:17.244957 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:17.244928    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-5c109b-55dfb5db7-2zxvt"]
Nov 03 01:17:17.269668 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:17.269643    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"]
Nov 03 01:17:17.273857 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:17.273826    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-5c109b-6c8b4fbc45-9nc8g"]
Nov 03 01:17:18.565706 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:18.565674    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e39d791-541f-4cc3-baec-5635b01020a6" path="/var/lib/kubelet/pods/1e39d791-541f-4cc3-baec-5635b01020a6/volumes"
Nov 03 01:17:18.566103 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:18.566035    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" path="/var/lib/kubelet/pods/caac0b46-9a05-41c3-8c6a-cf9097d5cf37/volumes"
Nov 03 01:17:19.553491 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.553455    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main" containerID="cri-o://0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89" gracePeriod=45
Nov 03 01:17:19.565660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.565624    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main" containerID="cri-o://62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193" gracePeriod=45
Nov 03 01:17:19.812980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.812914    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 01:17:19.838352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.838330    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 01:17:19.901972 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:19.901922    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:17:19.904103 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.904076    6000 generic.go:356] "Generic (PLEG): container finished" podID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerID="62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193" exitCode=0
Nov 03 01:17:19.904288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.904118    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"
Nov 03 01:17:19.904288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.904160    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" event={"ID":"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca","Type":"ContainerDied","Data":"62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193"}
Nov 03 01:17:19.904288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.904208    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl" event={"ID":"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca","Type":"ContainerDied","Data":"7d5c720eb70f13cb5d1e57b4594a57e025b1158438cc6d916d533ba01e7ca096"}
Nov 03 01:17:19.904288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.904232    6000 scope.go:122] "RemoveContainer" containerID="62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193"
Nov 03 01:17:19.906502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.906465    6000 generic.go:356] "Generic (PLEG): container finished" podID="ea440629-1449-4a15-8565-304fb54877ce" containerID="0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89" exitCode=0
Nov 03 01:17:19.906642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.906505    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"
Nov 03 01:17:19.906642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.906510    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" event={"ID":"ea440629-1449-4a15-8565-304fb54877ce","Type":"ContainerDied","Data":"0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89"}
Nov 03 01:17:19.906642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.906540    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76" event={"ID":"ea440629-1449-4a15-8565-304fb54877ce","Type":"ContainerDied","Data":"5af684ea7d39b7fbe994f0322cd0049a6fceeaf0e04bb66a002b30acebfcc5d1"}
Nov 03 01:17:19.915394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.915366    6000 scope.go:122] "RemoveContainer" containerID="62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193"
Nov 03 01:17:19.915724 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:19.915702    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193\": container with ID starting with 62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193 not found: ID does not exist" containerID="62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193"
Nov 03 01:17:19.915840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.915730    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193"} err="failed to get container status \"62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193\": rpc error: code = NotFound desc = could not find container \"62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193\": container with ID starting with 62f6d55350716005d5ab2238a21d66dccb0128d74ff021d57c8e82eb34e7f193 not found: ID does not exist"
Nov 03 01:17:19.915840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.915751    6000 scope.go:122] "RemoveContainer" containerID="0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89"
Nov 03 01:17:19.918033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918005    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-home\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-home\") pod \"ea440629-1449-4a15-8565-304fb54877ce\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") "
Nov 03 01:17:19.918139 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918050    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea440629-1449-4a15-8565-304fb54877ce-kube-api-access-p6554\" (UniqueName: \"kubernetes.io/projected/ea440629-1449-4a15-8565-304fb54877ce-kube-api-access-p6554\") pod \"ea440629-1449-4a15-8565-304fb54877ce\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") "
Nov 03 01:17:19.918139 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918118    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea440629-1449-4a15-8565-304fb54877ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440629-1449-4a15-8565-304fb54877ce-tls-certs\") pod \"ea440629-1449-4a15-8565-304fb54877ce\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") "
Nov 03 01:17:19.918312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918162    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-tmp-dir\") pod \"ea440629-1449-4a15-8565-304fb54877ce\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") "
Nov 03 01:17:19.918312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918197    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-model-cache\") pod \"ea440629-1449-4a15-8565-304fb54877ce\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") "
Nov 03 01:17:19.918312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918246    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tmp-dir\") pod \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") "
Nov 03 01:17:19.918312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918295    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-home\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-home\") pod \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") "
Nov 03 01:17:19.918572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918320    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-home" pod "ea440629-1449-4a15-8565-304fb54877ce" (UID: "ea440629-1449-4a15-8565-304fb54877ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.918572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918331    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-model-cache\") pod \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") "
Nov 03 01:17:19.918572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918399    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tls-certs\" (UniqueName: \"kubernetes.io/secret/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tls-certs\") pod \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") "
Nov 03 01:17:19.918572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918448    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-dshm\") pod \"ea440629-1449-4a15-8565-304fb54877ce\" (UID: \"ea440629-1449-4a15-8565-304fb54877ce\") "
Nov 03 01:17:19.918572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918492    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-dshm\") pod \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") "
Nov 03 01:17:19.918572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918564    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-model-cache" pod "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" (UID: "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.918924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918578    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-kube-api-access-9fmhh\" (UniqueName: \"kubernetes.io/projected/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-kube-api-access-9fmhh\") pod \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\" (UID: \"d460a3cf-16dd-49ad-a3d8-05ab4a8112ca\") "
Nov 03 01:17:19.918924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918763    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tmp-dir" pod "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" (UID: "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.919031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.918920    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-home" pod "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" (UID: "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.919031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.919011    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.919031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.919029    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.919213 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.919043    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.919213 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.919061    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:19.919213 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.919144    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-tmp-dir" pod "ea440629-1449-4a15-8565-304fb54877ce" (UID: "ea440629-1449-4a15-8565-304fb54877ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.919430 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.919213    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-model-cache" pod "ea440629-1449-4a15-8565-304fb54877ce" (UID: "ea440629-1449-4a15-8565-304fb54877ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.922134 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.922102    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea440629-1449-4a15-8565-304fb54877ce-tls-certs" pod "ea440629-1449-4a15-8565-304fb54877ce" (UID: "ea440629-1449-4a15-8565-304fb54877ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:19.922589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.922559    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-kube-api-access-9fmhh" pod "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" (UID: "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca"). InnerVolumeSpecName "kube-api-access-9fmhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:19.922589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.922577    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea440629-1449-4a15-8565-304fb54877ce-kube-api-access-p6554" pod "ea440629-1449-4a15-8565-304fb54877ce" (UID: "ea440629-1449-4a15-8565-304fb54877ce"). InnerVolumeSpecName "kube-api-access-p6554". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:19.922780 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.922588    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-dshm" pod "ea440629-1449-4a15-8565-304fb54877ce" (UID: "ea440629-1449-4a15-8565-304fb54877ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.922924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.922900    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tls-certs" pod "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" (UID: "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:19.924530 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.924505    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-dshm" pod "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" (UID: "d460a3cf-16dd-49ad-a3d8-05ab4a8112ca"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:19.926438 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.926417    6000 scope.go:122] "RemoveContainer" containerID="0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89"
Nov 03 01:17:19.926717 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:19.926698    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89\": container with ID starting with 0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89 not found: ID does not exist" containerID="0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89"
Nov 03 01:17:19.926808 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:19.926725    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89"} err="failed to get container status \"0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89\": rpc error: code = NotFound desc = could not find container \"0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89\": container with ID starting with 0931c33d305b3cb897193c063ce62b16826e3f762e3a088b3284fc2259b3fa89 not found: ID does not exist"
Nov 03 01:17:20.020595 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020558    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea440629-1449-4a15-8565-304fb54877ce-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.020595 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020589    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.020595 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020602    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.020846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020613    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.020846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020625    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea440629-1449-4a15-8565-304fb54877ce-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.020846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020635    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.020846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020646    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fmhh\" (UniqueName: \"kubernetes.io/projected/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca-kube-api-access-9fmhh\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.020846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.020658    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6554\" (UniqueName: \"kubernetes.io/projected/ea440629-1449-4a15-8565-304fb54877ce-kube-api-access-p6554\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:20.273467 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.273432    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"]
Nov 03 01:17:20.285801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.285771    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-w1-b85343-kserve-9d8dd6cf9-4rptl"]
Nov 03 01:17:20.337950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.337924    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"]
Nov 03 01:17:20.344890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.344868    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-b85343/distinct-2-w1-b85343-kserve-ff7d9d99b-nwn76"]
Nov 03 01:17:20.565662 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.565591    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" path="/var/lib/kubelet/pods/d460a3cf-16dd-49ad-a3d8-05ab4a8112ca/volumes"
Nov 03 01:17:20.565993 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:20.565981    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea440629-1449-4a15-8565-304fb54877ce" path="/var/lib/kubelet/pods/ea440629-1449-4a15-8565-304fb54877ce/volumes"
Nov 03 01:17:24.320620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.320583    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"]
Nov 03 01:17:24.321171 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.320818    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x" podUID="e886baa1-3552-4ee4-8609-13c51431268d" containerName="maas-api" containerID="cri-o://7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a" gracePeriod=30
Nov 03 01:17:24.571220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.571194    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 01:17:24.663410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.663380    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-maas-api-tls\") pod \"e886baa1-3552-4ee4-8609-13c51431268d\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") "
Nov 03 01:17:24.663619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.663505    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-metrics-certs\") pod \"e886baa1-3552-4ee4-8609-13c51431268d\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") "
Nov 03 01:17:24.663619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.663558    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e886baa1-3552-4ee4-8609-13c51431268d-kube-api-access-xgzdf\" (UniqueName: \"kubernetes.io/projected/e886baa1-3552-4ee4-8609-13c51431268d-kube-api-access-xgzdf\") pod \"e886baa1-3552-4ee4-8609-13c51431268d\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") "
Nov 03 01:17:24.663756 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.663637    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e886baa1-3552-4ee4-8609-13c51431268d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886baa1-3552-4ee4-8609-13c51431268d-openshift-service-ca\") pod \"e886baa1-3552-4ee4-8609-13c51431268d\" (UID: \"e886baa1-3552-4ee4-8609-13c51431268d\") "
Nov 03 01:17:24.664200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.664061    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e886baa1-3552-4ee4-8609-13c51431268d-openshift-service-ca" pod "e886baa1-3552-4ee4-8609-13c51431268d" (UID: "e886baa1-3552-4ee4-8609-13c51431268d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:24.665784 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.665757    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-metrics-certs" pod "e886baa1-3552-4ee4-8609-13c51431268d" (UID: "e886baa1-3552-4ee4-8609-13c51431268d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:24.665909 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.665807    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-maas-api-tls" pod "e886baa1-3552-4ee4-8609-13c51431268d" (UID: "e886baa1-3552-4ee4-8609-13c51431268d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:17:24.665909 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.665892    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e886baa1-3552-4ee4-8609-13c51431268d-kube-api-access-xgzdf" pod "e886baa1-3552-4ee4-8609-13c51431268d" (UID: "e886baa1-3552-4ee4-8609-13c51431268d"). InnerVolumeSpecName "kube-api-access-xgzdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:24.764746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.764654    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e886baa1-3552-4ee4-8609-13c51431268d-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:24.764746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.764685    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:24.764746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.764694    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e886baa1-3552-4ee4-8609-13c51431268d-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:24.764746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.764703    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xgzdf\" (UniqueName: \"kubernetes.io/projected/e886baa1-3552-4ee4-8609-13c51431268d-kube-api-access-xgzdf\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:24.932655 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.932619    6000 generic.go:356] "Generic (PLEG): container finished" podID="e886baa1-3552-4ee4-8609-13c51431268d" containerID="7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a" exitCode=0
Nov 03 01:17:24.932973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.932664    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"
Nov 03 01:17:24.932973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.932703    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x" event={"ID":"e886baa1-3552-4ee4-8609-13c51431268d","Type":"ContainerDied","Data":"7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a"}
Nov 03 01:17:24.932973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.932760    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x" event={"ID":"e886baa1-3552-4ee4-8609-13c51431268d","Type":"ContainerDied","Data":"c42efae3511169674abae7c1e249b2d3f6f6780c7ca7b2b126516a37b5fb7bf2"}
Nov 03 01:17:24.932973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.932790    6000 scope.go:122] "RemoveContainer" containerID="7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a"
Nov 03 01:17:24.942396 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.942376    6000 scope.go:122] "RemoveContainer" containerID="7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a"
Nov 03 01:17:24.942665 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:24.942641    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a\": container with ID starting with 7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a not found: ID does not exist" containerID="7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a"
Nov 03 01:17:24.942717 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.942678    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a"} err="failed to get container status \"7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a\": rpc error: code = NotFound desc = could not find container \"7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a\": container with ID starting with 7a204d417e373f37240e34f0bb2ac03c31888ae2f64580fa151b3585612b1e2a not found: ID does not exist"
Nov 03 01:17:24.980131 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.980104    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"]
Nov 03 01:17:24.984844 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:24.984821    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-b85343-dff9f88bf-qqk5x"]
Nov 03 01:17:26.565804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:26.565762    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e886baa1-3552-4ee4-8609-13c51431268d" path="/var/lib/kubelet/pods/e886baa1-3552-4ee4-8609-13c51431268d/volumes"
Nov 03 01:17:27.651486 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.651445    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:17:27.977329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977209    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"]
Nov 03 01:17:27.977624 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977608    6000 cpu_manager.go:414] "containerMap: removing container" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy"
Nov 03 01:17:27.977624 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977624    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977632    6000 cpu_manager.go:414] "containerMap: removing container" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977638    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977646    6000 cpu_manager.go:414] "containerMap: removing container" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977651    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977663    6000 cpu_manager.go:414] "containerMap: removing container" podUID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" containerName="payload-pre-processing"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977669    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" containerName="payload-pre-processing"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977676    6000 cpu_manager.go:414] "containerMap: removing container" podUID="54d2cebd-68c0-4b15-ac8e-5088762e5cdf" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977683    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54d2cebd-68c0-4b15-ac8e-5088762e5cdf" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977696    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977702    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977713    6000 cpu_manager.go:414] "containerMap: removing container" podUID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" containerName="payload-pre-processing"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977719    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" containerName="payload-pre-processing"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977733    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977738    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977744    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977749    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977757    6000 cpu_manager.go:414] "containerMap: removing container" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977763    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977769    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977773    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977780    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977785    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" containerName="main"
Nov 03 01:17:27.977778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977794    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1e39d791-541f-4cc3-baec-5635b01020a6" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977798    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e39d791-541f-4cc3-baec-5635b01020a6" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977804    6000 cpu_manager.go:414] "containerMap: removing container" podUID="0fd57543-8daa-4be4-a9f9-0208bcd55b4e" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977808    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd57543-8daa-4be4-a9f9-0208bcd55b4e" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977816    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c37bf3b6-0b50-4f34-837b-15368f5d6de7" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977821    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c37bf3b6-0b50-4f34-837b-15368f5d6de7" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977827    6000 cpu_manager.go:414] "containerMap: removing container" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977832    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977844    6000 cpu_manager.go:414] "containerMap: removing container" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977851    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977862    6000 cpu_manager.go:414] "containerMap: removing container" podUID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977870    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977882    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7c7b20a3-0524-450e-b61b-c69543af4191" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977890    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c7b20a3-0524-450e-b61b-c69543af4191" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977901    6000 cpu_manager.go:414] "containerMap: removing container" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977907    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977919    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977923    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977932    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977936    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977943    6000 cpu_manager.go:414] "containerMap: removing container" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977947    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977954    6000 cpu_manager.go:414] "containerMap: removing container" podUID="66432f27-b3ec-443d-a298-9c1aea8b27f4" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977958    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66432f27-b3ec-443d-a298-9c1aea8b27f4" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977964    6000 cpu_manager.go:414] "containerMap: removing container" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977970    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977981    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977986    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977991    6000 cpu_manager.go:414] "containerMap: removing container" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.977997    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978009    6000 cpu_manager.go:414] "containerMap: removing container" podUID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978013    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978021    6000 cpu_manager.go:414] "containerMap: removing container" podUID="de14e2d3-979f-4186-91fe-6f8630291d8a" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978027    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de14e2d3-979f-4186-91fe-6f8630291d8a" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978038    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978044    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978050    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978055    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978065    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978072    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978084    6000 cpu_manager.go:414] "containerMap: removing container" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978091    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978101    6000 cpu_manager.go:414] "containerMap: removing container" podUID="f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978106    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978131    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978137    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978150    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e886baa1-3552-4ee4-8609-13c51431268d" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978157    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e886baa1-3552-4ee4-8609-13c51431268d" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978166    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978170    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978178    6000 cpu_manager.go:414] "containerMap: removing container" podUID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978183    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978189    6000 cpu_manager.go:414] "containerMap: removing container" podUID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978196    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978205    6000 cpu_manager.go:414] "containerMap: removing container" podUID="c746ad54-d687-422d-b0f4-3ba0e6f31a55" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978211    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c746ad54-d687-422d-b0f4-3ba0e6f31a55" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978221    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7fa21b75-6104-431b-ba39-a56035f517ac" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978227    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa21b75-6104-431b-ba39-a56035f517ac" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978235    6000 cpu_manager.go:414] "containerMap: removing container" podUID="619a36e0-3a99-47b1-ae3a-0a9395e118a4" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978241    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619a36e0-3a99-47b1-ae3a-0a9395e118a4" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978337    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978349    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978363    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cbb9852-0c50-49c4-bb1b-631b6c394e8b" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978375    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="de14e2d3-979f-4186-91fe-6f8630291d8a" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978383    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2dde3ef-cadd-4c34-a7ef-ca4b27817ff6" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978389    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978395    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="09ab309f-7e1a-4789-b54e-f609b6c94ad6" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978405    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1ca0eaa-8944-45a6-bc7f-5da1d50b62a8" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978412    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978417    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e886baa1-3552-4ee4-8609-13c51431268d" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978425    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978434    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="de6eba37-c7cd-4239-a7fd-9b815b14dc79" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978440    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="3adb0154-70eb-464b-bfcb-3ffc3d5476e0" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978449    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c746ad54-d687-422d-b0f4-3ba0e6f31a55" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978458    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa21b75-6104-431b-ba39-a56035f517ac" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978466    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="619a36e0-3a99-47b1-ae3a-0a9395e118a4" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978471    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd96e482-f9fd-4acb-b4eb-195f6c98d526" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978479    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978487    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978492    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="caac0b46-9a05-41c3-8c6a-cf9097d5cf37" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978499    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="54d2cebd-68c0-4b15-ac8e-5088762e5cdf" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978505    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d460a3cf-16dd-49ad-a3d8-05ab4a8112ca" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978511    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0f400e1-4d6e-41e3-8dd9-7368268d6a4f" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978516    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978521    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="02281a18-5373-45a1-89a6-25f6ca6e2911" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978526    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea440629-1449-4a15-8565-304fb54877ce" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978533    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="f985caf0-f2c4-4f84-b3e8-b47f4a5368e7" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978541    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e39d791-541f-4cc3-baec-5635b01020a6" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978552    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd57543-8daa-4be4-a9f9-0208bcd55b4e" containerName="maas-api"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978560    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="c37bf3b6-0b50-4f34-837b-15368f5d6de7" containerName="payload-pre-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978568    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978574    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f9048e-4dfe-4376-90e6-58c49ff9fc8c" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978580    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c7b20a3-0524-450e-b61b-c69543af4191" containerName="payload-processing"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978586    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="14b40ec4-3b82-4726-a95b-f80577c4b051" containerName="istio-proxy"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978596    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978601    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="66432f27-b3ec-443d-a298-9c1aea8b27f4" containerName="main"
Nov 03 01:17:27.979209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.978607    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:27.983716 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.983697    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:27.997494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:27.997471    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 01:17:28.008716 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.008687    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-99bdb8-openshift-default-dockercfg-dq2f2\""
Nov 03 01:17:28.010420 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.010397    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"]
Nov 03 01:17:28.094242 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094206    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094439 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094245    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094439 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094400    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094526 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094456    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmg8\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-kube-api-access-swmg8\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094526 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094491    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094604 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094525    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094604 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094589    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595f051f-3ae5-422b-b1bd-a019b6734ccd-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094674 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094650    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.094712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.094690    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195552 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195517    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195569    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595f051f-3ae5-422b-b1bd-a019b6734ccd-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195595    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195631    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195662    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195687    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195735    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195772    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmg8\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-kube-api-access-swmg8\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.195973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195849    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.196148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.195997    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.196212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.196158    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-credential-socket\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.196316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.196239    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-data\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.196462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.196429    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595f051f-3ae5-422b-b1bd-a019b6734ccd-istiod-ca-cert\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.196587 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.196413    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-certs\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.198165 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.198144    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-envoy\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.198394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.198373    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-podinfo\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.213924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.213902    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-token\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.214152 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.214128    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmg8\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-kube-api-access-swmg8\") pod \"e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") " pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.299140 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.299055    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:28.606484 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.606460    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"]
Nov 03 01:17:28.608722 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:17:28.608697    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod595f051f_3ae5_422b_b1bd_a019b6734ccd.slice/crio-d434414818cc73b01ecf24bf9220d2b1b87860b1ef6ccf012f2327d2c332cd40 WatchSource:0}: Error finding container d434414818cc73b01ecf24bf9220d2b1b87860b1ef6ccf012f2327d2c332cd40: Status 404 returned error can't find the container with id d434414818cc73b01ecf24bf9220d2b1b87860b1ef6ccf012f2327d2c332cd40
Nov 03 01:17:28.610837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.610801    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:17:28.610981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.610887    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:17:28.610981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.610933    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:17:28.953102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.953017    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" event={"ID":"595f051f-3ae5-422b-b1bd-a019b6734ccd","Type":"ContainerStarted","Data":"1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5"}
Nov 03 01:17:28.953102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:28.953074    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" event={"ID":"595f051f-3ae5-422b-b1bd-a019b6734ccd","Type":"ContainerStarted","Data":"d434414818cc73b01ecf24bf9220d2b1b87860b1ef6ccf012f2327d2c332cd40"}
Nov 03 01:17:29.015308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:29.015228    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" podStartSLOduration=2.015213421 podStartE2EDuration="2.015213421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:29.003523993 +0000 UTC m=+4286.558876017" watchObservedRunningTime="2026-11-03 01:17:29.015213421 +0000 UTC m=+4286.570565797"
Nov 03 01:17:29.307341 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:29.307226    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:29.307617 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:29.307381    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 01:17:29.307775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:29.307753    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 01:17:30.308280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:30.306868    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 01:17:30.308280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:30.306941    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 01:17:31.314281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:31.301819    6000 patch_prober.go:29] interesting pod/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused" start-of-body=
Nov 03 01:17:31.314281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:31.301896    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.46:15021/healthz/ready\": dial tcp 10.134.1.46:15021: connect: connection refused"
Nov 03 01:17:32.077518 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:32.077486    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"]
Nov 03 01:17:32.077743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:32.077697    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerName="istio-proxy" containerID="cri-o://1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5" gracePeriod=30
Nov 03 01:17:40.238124 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.238095    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:40.318885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.318796    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-token\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-token\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.318885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.318860    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-socket\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319123 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.318916    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-credential-socket\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319123 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.318944    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/595f051f-3ae5-422b-b1bd-a019b6734ccd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595f051f-3ae5-422b-b1bd-a019b6734ccd-istiod-ca-cert\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319123 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.318993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-envoy\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319123 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319018    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-kube-api-access-swmg8\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-kube-api-access-swmg8\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319123 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319084    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-data\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319138    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-podinfo\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319165    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-certs\") pod \"595f051f-3ae5-422b-b1bd-a019b6734ccd\" (UID: \"595f051f-3ae5-422b-b1bd-a019b6734ccd\") "
Nov 03 01:17:40.319441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319191    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-socket" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.319441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319341    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-credential-socket" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.319677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319471    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-data" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.319737 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319686    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.319737 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319708    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.319737 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319723    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.320068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319763    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/595f051f-3ae5-422b-b1bd-a019b6734ccd-istiod-ca-cert" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:40.320068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.319920    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-certs" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.321562 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.321529    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-podinfo" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:40.321738 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.321719    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-kube-api-access-swmg8" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "kube-api-access-swmg8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:40.321856 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.321820    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-envoy" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:40.321978 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.321857    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-token" pod "595f051f-3ae5-422b-b1bd-a019b6734ccd" (UID: "595f051f-3ae5-422b-b1bd-a019b6734ccd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:40.420765 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.420733    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.420765 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.420761    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swmg8\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-kube-api-access-swmg8\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.420765 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.420771    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.421019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.420781    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/595f051f-3ae5-422b-b1bd-a019b6734ccd-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.421019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.420788    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/595f051f-3ae5-422b-b1bd-a019b6734ccd-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:40.421019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:40.420796    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/595f051f-3ae5-422b-b1bd-a019b6734ccd-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:41.012687 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.012644    6000 generic.go:356] "Generic (PLEG): container finished" podID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerID="1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5" exitCode=0
Nov 03 01:17:41.012871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.012709    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"
Nov 03 01:17:41.012871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.012735    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" event={"ID":"595f051f-3ae5-422b-b1bd-a019b6734ccd","Type":"ContainerDied","Data":"1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5"}
Nov 03 01:17:41.012871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.012784    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf" event={"ID":"595f051f-3ae5-422b-b1bd-a019b6734ccd","Type":"ContainerDied","Data":"d434414818cc73b01ecf24bf9220d2b1b87860b1ef6ccf012f2327d2c332cd40"}
Nov 03 01:17:41.012871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.012805    6000 scope.go:122] "RemoveContainer" containerID="1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5"
Nov 03 01:17:41.023594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.023575    6000 scope.go:122] "RemoveContainer" containerID="1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5"
Nov 03 01:17:41.023891 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:41.023868    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5\": container with ID starting with 1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5 not found: ID does not exist" containerID="1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5"
Nov 03 01:17:41.023974 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.023900    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5"} err="failed to get container status \"1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5\": rpc error: code = NotFound desc = could not find container \"1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5\": container with ID starting with 1bf04df5a6d015a5c194b2cb5e51e825ac789f9b2be24e23fa21b084632d8af5 not found: ID does not exist"
Nov 03 01:17:41.064720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.064669    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"]
Nov 03 01:17:41.069660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:41.069633    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-99bdb8-openshift-default-6d57d84b85-bj5nf"]
Nov 03 01:17:42.564732 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:42.564703    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" path="/var/lib/kubelet/pods/595f051f-3ae5-422b-b1bd-a019b6734ccd/volumes"
Nov 03 01:17:43.413211 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413180    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"]
Nov 03 01:17:43.413609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413596    6000 cpu_manager.go:414] "containerMap: removing container" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerName="istio-proxy"
Nov 03 01:17:43.413666 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413610    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerName="istio-proxy"
Nov 03 01:17:43.413708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413692    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7551f80e-1fa7-4afd-acf5-f0364d517ac6" containerName="cleanup"
Nov 03 01:17:43.413708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413704    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="595f051f-3ae5-422b-b1bd-a019b6734ccd" containerName="istio-proxy"
Nov 03 01:17:43.413797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413711    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="991686e6-13bc-41c3-a577-baf8686c1109" containerName="cleanup"
Nov 03 01:17:43.413797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413718    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba47343d-7e39-4e7b-855a-04edb08f8ba1" containerName="cleanup"
Nov 03 01:17:43.413797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413725    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4d39152-3fe9-47bd-b166-9a84ac7a5f98" containerName="istio-proxy"
Nov 03 01:17:43.413797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.413734    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="508ab7fa-8fe3-4390-8a38-c40d94180c3e" containerName="cleanup"
Nov 03 01:17:43.418175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.418150    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.432146 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.432125    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 01:17:43.443384 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.443358    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-b85343-openshift-default-dockercfg-jnjww\""
Nov 03 01:17:43.445050 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.445015    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"]
Nov 03 01:17:43.552385 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552344    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552402    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552493    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552513    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552592    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552623    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552697 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552663    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tptj4\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-kube-api-access-tptj4\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552798 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552725    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.552798 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.552754    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f53be-7080-4c16-a780-acb57b98d1ed-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654270 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654221    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tptj4\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-kube-api-access-tptj4\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654279    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654311    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f53be-7080-4c16-a780-acb57b98d1ed-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654355    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654400    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654432    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654460    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654493    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654520    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.654796 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654789    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-data\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.655278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.654961    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.655278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.655002    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-credential-socket\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.655278 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.655017    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-certs\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.655414 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.655363    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f53be-7080-4c16-a780-acb57b98d1ed-istiod-ca-cert\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.657045 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.657020    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-envoy\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.657546 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.657521    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-podinfo\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.673520 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.673453    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-token\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.673646 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.673526    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tptj4\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-kube-api-access-tptj4\") pod \"e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") " pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:43.731309 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:43.731271    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:44.038930 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:44.038835    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"]
Nov 03 01:17:44.044030 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:17:44.044000    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6f53be_7080_4c16_a780_acb57b98d1ed.slice/crio-c46ead4372552ca5fae7303f0d521b008af0fe180a8f1dc05969d668a61e3909 WatchSource:0}: Error finding container c46ead4372552ca5fae7303f0d521b008af0fe180a8f1dc05969d668a61e3909: Status 404 returned error can't find the container with id c46ead4372552ca5fae7303f0d521b008af0fe180a8f1dc05969d668a61e3909
Nov 03 01:17:44.046715 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:44.046684    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:17:44.046797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:44.046756    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:17:44.046797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:44.046786    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:17:45.036331 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:45.036283    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj" event={"ID":"cf6f53be-7080-4c16-a780-acb57b98d1ed","Type":"ContainerStarted","Data":"8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960"}
Nov 03 01:17:45.036871 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:45.036357    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj" event={"ID":"cf6f53be-7080-4c16-a780-acb57b98d1ed","Type":"ContainerStarted","Data":"c46ead4372552ca5fae7303f0d521b008af0fe180a8f1dc05969d668a61e3909"}
Nov 03 01:17:45.076985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:45.076914    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj" podStartSLOduration=2.076894248 podStartE2EDuration="2.076894248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:17:45.075704211 +0000 UTC m=+4302.631056245" watchObservedRunningTime="2026-11-03 01:17:45.076894248 +0000 UTC m=+4302.632246273"
Nov 03 01:17:45.158649 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:45.149638    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"]
Nov 03 01:17:45.731949 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:45.731910    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:45.737092 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:45.737064    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:46.041064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:46.040981    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:46.041064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:46.041002    6000 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj" secret="" err="secret \"e2e-worker-w1-b85343-openshift-default-dockercfg-jnjww\" not found"
Nov 03 01:17:46.042137 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:46.042116    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:47.045859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:47.045797    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj" podUID="cf6f53be-7080-4c16-a780-acb57b98d1ed" containerName="istio-proxy" containerID="cri-o://8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960" gracePeriod=30
Nov 03 01:17:47.847603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:47.847571    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kxq2l_f109704e-d9c1-41e3-84c9-5a0aa3eac6be/limitador/0.log"
Nov 03 01:17:49.193352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:49.193322    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77bbb667f4-ps8vw_ec8b5850-10d2-4e94-96fc-834119b90561/manager/0.log"
Nov 03 01:17:49.896470 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:49.896429    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:17:50.376803 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:50.376757    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc8c7d7fb-9crlh_65984e3f-0d1a-4d8c-ac3a-ca76eda948a5/manager/1.log"
Nov 03 01:17:51.005980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:51.005952    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hm874_6f731881-a91d-48e5-b881-8bee6561c7fb/proxy/0.log"
Nov 03 01:17:51.611102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:51.611077    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc8c7d7fb-9crlh_65984e3f-0d1a-4d8c-ac3a-ca76eda948a5/manager/0.log"
Nov 03 01:17:52.297569 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.297545    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:52.439109 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439055    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-podinfo\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439109 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439109    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-envoy\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439109 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439128    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-certs\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439246    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-socket\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439323    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6f53be-7080-4c16-a780-acb57b98d1ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f53be-7080-4c16-a780-acb57b98d1ed-istiod-ca-cert\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439372    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-credential-socket\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439403    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-kube-api-access-tptj4\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-kube-api-access-tptj4\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439463    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-token\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439478    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-certs" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:52.439494 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439490    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-data\") pod \"cf6f53be-7080-4c16-a780-acb57b98d1ed\" (UID: \"cf6f53be-7080-4c16-a780-acb57b98d1ed\") "
Nov 03 01:17:52.439883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439515    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-socket" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:52.439883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439597    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-credential-socket" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:52.439883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439702    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6f53be-7080-4c16-a780-acb57b98d1ed-istiod-ca-cert" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:17:52.439883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.439875    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-data" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:52.440308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.440131    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6f53be-7080-4c16-a780-acb57b98d1ed-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.440308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.440152    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.440308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.440167    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.440308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.440183    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.440308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.440195    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.441528 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.441502    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-podinfo" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:17:52.441690 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.441673    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-token" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:52.441739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.441700    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-kube-api-access-tptj4" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "kube-api-access-tptj4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:17:52.441739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.441726    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-envoy" pod "cf6f53be-7080-4c16-a780-acb57b98d1ed" (UID: "cf6f53be-7080-4c16-a780-acb57b98d1ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:17:52.540694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.540663    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.540694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.540687    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.540694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.540697    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6f53be-7080-4c16-a780-acb57b98d1ed-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:52.540924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:52.540705    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tptj4\" (UniqueName: \"kubernetes.io/projected/cf6f53be-7080-4c16-a780-acb57b98d1ed-kube-api-access-tptj4\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:17:53.074869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.074836    6000 generic.go:356] "Generic (PLEG): container finished" podID="cf6f53be-7080-4c16-a780-acb57b98d1ed" containerID="8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960" exitCode=0
Nov 03 01:17:53.075337 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.074941    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"
Nov 03 01:17:53.075337 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.074973    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj" event={"ID":"cf6f53be-7080-4c16-a780-acb57b98d1ed","Type":"ContainerDied","Data":"8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960"}
Nov 03 01:17:53.075337 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.075026    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj" event={"ID":"cf6f53be-7080-4c16-a780-acb57b98d1ed","Type":"ContainerDied","Data":"c46ead4372552ca5fae7303f0d521b008af0fe180a8f1dc05969d668a61e3909"}
Nov 03 01:17:53.075337 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.075049    6000 scope.go:122] "RemoveContainer" containerID="8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960"
Nov 03 01:17:53.083865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.083845    6000 scope.go:122] "RemoveContainer" containerID="8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960"
Nov 03 01:17:53.084127 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:17:53.084106    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960\": container with ID starting with 8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960 not found: ID does not exist" containerID="8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960"
Nov 03 01:17:53.084202 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.084136    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960"} err="failed to get container status \"8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960\": rpc error: code = NotFound desc = could not find container \"8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960\": container with ID starting with 8f18eb623d4143e1bca0e82ba81e7d4f95e4ab3da29cfcc7773eea96ef395960 not found: ID does not exist"
Nov 03 01:17:53.135330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.135292    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"]
Nov 03 01:17:53.146527 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:53.146497    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-b85343-openshift-default-7f5cd764b9-7pfjj"]
Nov 03 01:17:54.565295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:17:54.565242    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6f53be-7080-4c16-a780-acb57b98d1ed" path="/var/lib/kubelet/pods/cf6f53be-7080-4c16-a780-acb57b98d1ed/volumes"
Nov 03 01:18:00.745892 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.745859    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"]
Nov 03 01:18:00.746507 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.746485    6000 cpu_manager.go:414] "containerMap: removing container" podUID="cf6f53be-7080-4c16-a780-acb57b98d1ed" containerName="istio-proxy"
Nov 03 01:18:00.746564 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.746511    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6f53be-7080-4c16-a780-acb57b98d1ed" containerName="istio-proxy"
Nov 03 01:18:00.746647 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.746634    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6f53be-7080-4c16-a780-acb57b98d1ed" containerName="istio-proxy"
Nov 03 01:18:00.751396 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.751368    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.766925 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.766905    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 01:18:00.771536 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.771514    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-0e3bc4e3-openshift-default-dockercfg-rmzr6\""
Nov 03 01:18:00.783394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.783368    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"]
Nov 03 01:18:00.814619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.814585    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-envoy\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.814785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.814640    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-credential-socket\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.814785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.814696    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f78ss\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-kube-api-access-f78ss\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.814926 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.814821    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-certs\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.814926 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.814866    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-socket\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.815048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.814985    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-podinfo\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.815048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.815022    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-data\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.815140 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.815117    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-token\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.815194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.815152    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istiod-ca-cert\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916009 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.915973    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-socket\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916032    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-podinfo\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916057    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-data\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916082    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-token\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916100    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istiod-ca-cert\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916126    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-envoy\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916174    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-credential-socket\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916197    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f78ss\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-kube-api-access-f78ss\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916627 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916242    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-certs\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916627 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916449    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-socket\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916627 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916615    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-data\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916652    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-credential-socket\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916717    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-certs\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.916996 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.916977    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istiod-ca-cert\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.918720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.918696    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-envoy\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.918973 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.918953    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-podinfo\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.938103 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.938078    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f78ss\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-kube-api-access-f78ss\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:00.938208 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:00.938154    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-token\") pod \"e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") " pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:01.066706 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:01.066607    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:01.382169 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:01.382140    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"]
Nov 03 01:18:01.383569 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:18:01.383539    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod069b60ab_cde7_4e2a_adf1_b391d1e9793e.slice/crio-5c5dc2e6a5e5e3f221233eafd5785a6d59c73f555e7e57d3d9a22c509cdc6138 WatchSource:0}: Error finding container 5c5dc2e6a5e5e3f221233eafd5785a6d59c73f555e7e57d3d9a22c509cdc6138: Status 404 returned error can't find the container with id 5c5dc2e6a5e5e3f221233eafd5785a6d59c73f555e7e57d3d9a22c509cdc6138
Nov 03 01:18:01.385668 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:01.385632    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:01.385790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:01.385709    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:01.385790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:01.385755    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:02.120318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:02.120278    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" event={"ID":"069b60ab-cde7-4e2a-adf1-b391d1e9793e","Type":"ContainerStarted","Data":"fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a"}
Nov 03 01:18:02.120789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:02.120331    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" event={"ID":"069b60ab-cde7-4e2a-adf1-b391d1e9793e","Type":"ContainerStarted","Data":"5c5dc2e6a5e5e3f221233eafd5785a6d59c73f555e7e57d3d9a22c509cdc6138"}
Nov 03 01:18:02.171101 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:02.171041    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podStartSLOduration=2.171024054 podStartE2EDuration="2.171024054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:02.170633716 +0000 UTC m=+4319.725985740" watchObservedRunningTime="2026-11-03 01:18:02.171024054 +0000 UTC m=+4319.726376078"
Nov 03 01:18:03.066840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:03.066804    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:03.071854 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:03.071828    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:03.126490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:03.126460    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:03.127373 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:03.127357    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:05.212681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.212644    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"]
Nov 03 01:18:05.216360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.216335    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.230059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.230039    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-0e3bc4e3-serving-cert\""
Nov 03 01:18:05.243621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.243595    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"]
Nov 03 01:18:05.258798 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.258769    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfjd9\" (UniqueName: \"kubernetes.io/projected/ee7795bd-0495-4d70-99bf-b57764a2cff4-kube-api-access-qfjd9\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.258958 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.258824    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.259220 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.259189    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7795bd-0495-4d70-99bf-b57764a2cff4-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.259365 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.259276    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-metrics-certs\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.360224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.360189    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7795bd-0495-4d70-99bf-b57764a2cff4-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.360444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.360235    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-metrics-certs\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.360444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.360289    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfjd9\" (UniqueName: \"kubernetes.io/projected/ee7795bd-0495-4d70-99bf-b57764a2cff4-kube-api-access-qfjd9\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.360444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.360318    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.360444 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:05.360422    6000 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-serving-cert: secret "maas-api-e2e-ait-0e3bc4e3-serving-cert" not found
Nov 03 01:18:05.360685 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:05.360478    6000 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls podName:ee7795bd-0495-4d70-99bf-b57764a2cff4 nodeName:}" failed. No retries permitted until 2026-11-03 01:18:05.860463719 +0000 UTC m=+4323.415815721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls") pod "maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd" (UID: "ee7795bd-0495-4d70-99bf-b57764a2cff4") : secret "maas-api-e2e-ait-0e3bc4e3-serving-cert" not found
Nov 03 01:18:05.360900 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.360879    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7795bd-0495-4d70-99bf-b57764a2cff4-openshift-service-ca\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.362858 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.362840    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-metrics-certs\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.379404 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.379378    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfjd9\" (UniqueName: \"kubernetes.io/projected/ee7795bd-0495-4d70-99bf-b57764a2cff4-kube-api-access-qfjd9\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.655751 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.655717    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"]
Nov 03 01:18:05.659418 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.659390    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.675842 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.675816    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e3bc4e3-dockercfg-5kvzx\""
Nov 03 01:18:05.676013 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.675887    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e3bc4e3\""
Nov 03 01:18:05.677410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.677379    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"]
Nov 03 01:18:05.745875 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.745845    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"]
Nov 03 01:18:05.749638 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.749616    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:05.759015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.758988    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"]
Nov 03 01:18:05.765982 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.765954    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e465bd-eed8-4e0b-b2cf-5fad31038064-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.766099 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.766025    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8bn\" (UniqueName: \"kubernetes.io/projected/75e465bd-eed8-4e0b-b2cf-5fad31038064-kube-api-access-hk8bn\") pod \"payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.866838 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.866804    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8bn\" (UniqueName: \"kubernetes.io/projected/75e465bd-eed8-4e0b-b2cf-5fad31038064-kube-api-access-hk8bn\") pod \"payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.867020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.866890    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a7131-ef7e-4ef2-96ac-985f7b89b19f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:05.867020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.866944    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdkxn\" (UniqueName: \"kubernetes.io/projected/654a7131-ef7e-4ef2-96ac-985f7b89b19f-kube-api-access-zdkxn\") pod \"payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:05.867020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.866980    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.867020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.867003    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e465bd-eed8-4e0b-b2cf-5fad31038064-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.867809 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.867780    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e465bd-eed8-4e0b-b2cf-5fad31038064-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.869576 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.869557    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls\") pod \"maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:05.885934 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.885910    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8bn\" (UniqueName: \"kubernetes.io/projected/75e465bd-eed8-4e0b-b2cf-5fad31038064-kube-api-access-hk8bn\") pod \"payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.968318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.968191    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a7131-ef7e-4ef2-96ac-985f7b89b19f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:05.968318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.968277    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdkxn\" (UniqueName: \"kubernetes.io/projected/654a7131-ef7e-4ef2-96ac-985f7b89b19f-kube-api-access-zdkxn\") pod \"payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:05.968995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.968968    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a7131-ef7e-4ef2-96ac-985f7b89b19f-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:05.973121 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.973093    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:05.987695 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:05.987670    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdkxn\" (UniqueName: \"kubernetes.io/projected/654a7131-ef7e-4ef2-96ac-985f7b89b19f-kube-api-access-zdkxn\") pod \"payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:06.062808 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:06.062779    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:06.129650 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:06.129617    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:06.267095 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:06.267008    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"]
Nov 03 01:18:06.271387 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:18:06.271359    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e465bd_eed8_4e0b_b2cf_5fad31038064.slice/crio-21f1e5ea8f5bcc668612a9dcd52f9c62ccc18d43f766d3561b782f2dd0b3d220 WatchSource:0}: Error finding container 21f1e5ea8f5bcc668612a9dcd52f9c62ccc18d43f766d3561b782f2dd0b3d220: Status 404 returned error can't find the container with id 21f1e5ea8f5bcc668612a9dcd52f9c62ccc18d43f766d3561b782f2dd0b3d220
Nov 03 01:18:06.356037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:06.356012    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"]
Nov 03 01:18:06.357137 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:18:06.357110    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod654a7131_ef7e_4ef2_96ac_985f7b89b19f.slice/crio-e9a204b2e56e75d03b3d00ca409f6c953c5f932ebc312f7e68fe8e488c327777 WatchSource:0}: Error finding container e9a204b2e56e75d03b3d00ca409f6c953c5f932ebc312f7e68fe8e488c327777: Status 404 returned error can't find the container with id e9a204b2e56e75d03b3d00ca409f6c953c5f932ebc312f7e68fe8e488c327777
Nov 03 01:18:06.429881 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:06.429750    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"]
Nov 03 01:18:06.432110 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:18:06.432079    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee7795bd_0495_4d70_99bf_b57764a2cff4.slice/crio-d8d057e4c6fc8231383bcae318c693075f7b4c3cbad50ed86767d503e569e57c WatchSource:0}: Error finding container d8d057e4c6fc8231383bcae318c693075f7b4c3cbad50ed86767d503e569e57c: Status 404 returned error can't find the container with id d8d057e4c6fc8231383bcae318c693075f7b4c3cbad50ed86767d503e569e57c
Nov 03 01:18:06.433786 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:06.433766    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:18:07.147912 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.147819    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r" event={"ID":"654a7131-ef7e-4ef2-96ac-985f7b89b19f","Type":"ContainerStarted","Data":"4c3ad4e49f4562f6d2160a165687112de1eba0117d238cdc94fc410f0e2f1fd8"}
Nov 03 01:18:07.147912 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.147879    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r" event={"ID":"654a7131-ef7e-4ef2-96ac-985f7b89b19f","Type":"ContainerStarted","Data":"e9a204b2e56e75d03b3d00ca409f6c953c5f932ebc312f7e68fe8e488c327777"}
Nov 03 01:18:07.148181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.148019    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:07.149517 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.149484    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd" event={"ID":"ee7795bd-0495-4d70-99bf-b57764a2cff4","Type":"ContainerStarted","Data":"755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006"}
Nov 03 01:18:07.149636 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.149541    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:07.149636 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.149565    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd" event={"ID":"ee7795bd-0495-4d70-99bf-b57764a2cff4","Type":"ContainerStarted","Data":"d8d057e4c6fc8231383bcae318c693075f7b4c3cbad50ed86767d503e569e57c"}
Nov 03 01:18:07.150913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.150890    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942" event={"ID":"75e465bd-eed8-4e0b-b2cf-5fad31038064","Type":"ContainerStarted","Data":"7cd8208476d56dc302cb6bf355d664369fda23f9d89e99412cb40171da4d4924"}
Nov 03 01:18:07.151015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.150932    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:07.151015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.150947    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942" event={"ID":"75e465bd-eed8-4e0b-b2cf-5fad31038064","Type":"ContainerStarted","Data":"21f1e5ea8f5bcc668612a9dcd52f9c62ccc18d43f766d3561b782f2dd0b3d220"}
Nov 03 01:18:07.184962 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.184916    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r" podStartSLOduration=2.184902729 podStartE2EDuration="2.184902729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:07.18364497 +0000 UTC m=+4324.738996994" watchObservedRunningTime="2026-11-03 01:18:07.184902729 +0000 UTC m=+4324.740254752"
Nov 03 01:18:07.221738 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.221686    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942" podStartSLOduration=2.221673486 podStartE2EDuration="2.221673486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:07.221344941 +0000 UTC m=+4324.776696966" watchObservedRunningTime="2026-11-03 01:18:07.221673486 +0000 UTC m=+4324.777025509"
Nov 03 01:18:07.259718 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:07.259665    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd" podStartSLOduration=1.832186643 podStartE2EDuration="2.259651298s" totalImagesPullingTime="427.464655ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:07.258449677 +0000 UTC m=+4324.813801715" watchObservedRunningTime="2026-11-03 01:18:07.259651298 +0000 UTC m=+4324.815003322"
Nov 03 01:18:08.949440 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:08.949401    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:18:13.165077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:13.164989    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:18.161895 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:18.161864    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:18.162444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:18.161945    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:18.626084 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:18.626053    6000 patch_prober.go:29] interesting pod/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": read tcp 10.134.0.2:39722->10.134.1.48:15021: read: connection reset by peer" start-of-body=
Nov 03 01:18:18.626293 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:18.626130    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": read tcp 10.134.0.2:39722->10.134.1.48:15021: read: connection reset by peer"
Nov 03 01:18:19.234764 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:19.234732    6000 generic.go:356] "Generic (PLEG): container finished" podID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerID="fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a" exitCode=0
Nov 03 01:18:19.235187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:19.234777    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" event={"ID":"069b60ab-cde7-4e2a-adf1-b391d1e9793e","Type":"ContainerDied","Data":"fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a"}
Nov 03 01:18:19.235187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:19.235117    6000 scope.go:122] "RemoveContainer" containerID="fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a"
Nov 03 01:18:19.236315 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:19.236282    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:19.236388 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:19.236362    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:19.236476 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:19.236393    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:19.925143 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:19.925100    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:18:20.242018 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:20.241931    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" event={"ID":"069b60ab-cde7-4e2a-adf1-b391d1e9793e","Type":"ContainerStarted","Data":"8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33"}
Nov 03 01:18:21.067497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:21.067468    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:21.067779 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:21.067753    6000 patch_prober.go:29] interesting pod/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body=
Nov 03 01:18:21.067849 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:21.067819    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused"
Nov 03 01:18:22.067769 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:22.067734    6000 patch_prober.go:29] interesting pod/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body=
Nov 03 01:18:22.068197 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:22.067787    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused"
Nov 03 01:18:23.067744 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:23.067701    6000 patch_prober.go:29] interesting pod/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body=
Nov 03 01:18:23.067945 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:23.067777    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused"
Nov 03 01:18:24.067612 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:24.067580    6000 patch_prober.go:29] interesting pod/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body=
Nov 03 01:18:24.067820 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:24.067637    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused"
Nov 03 01:18:25.067762 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:25.067716    6000 patch_prober.go:29] interesting pod/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body=
Nov 03 01:18:25.068223 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:25.067790    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused"
Nov 03 01:18:26.071948 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:26.071914    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:26.072441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:26.072174    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:26.072812 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:26.072797    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:30.127191 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:30.127154    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"]
Nov 03 01:18:30.159244 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:30.159213    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"]
Nov 03 01:18:32.854367 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:32.854323    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"]
Nov 03 01:18:32.854961 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:32.854548    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd" podUID="ee7795bd-0495-4d70-99bf-b57764a2cff4" containerName="maas-api" containerID="cri-o://755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006" gracePeriod=30
Nov 03 01:18:33.121391 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.121365    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:33.251190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.251159    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee7795bd-0495-4d70-99bf-b57764a2cff4-kube-api-access-qfjd9\" (UniqueName: \"kubernetes.io/projected/ee7795bd-0495-4d70-99bf-b57764a2cff4-kube-api-access-qfjd9\") pod \"ee7795bd-0495-4d70-99bf-b57764a2cff4\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") "
Nov 03 01:18:33.251412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.251206    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-metrics-certs\") pod \"ee7795bd-0495-4d70-99bf-b57764a2cff4\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") "
Nov 03 01:18:33.251412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.251324    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee7795bd-0495-4d70-99bf-b57764a2cff4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7795bd-0495-4d70-99bf-b57764a2cff4-openshift-service-ca\") pod \"ee7795bd-0495-4d70-99bf-b57764a2cff4\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") "
Nov 03 01:18:33.251412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.251388    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls\") pod \"ee7795bd-0495-4d70-99bf-b57764a2cff4\" (UID: \"ee7795bd-0495-4d70-99bf-b57764a2cff4\") "
Nov 03 01:18:33.251754 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.251726    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee7795bd-0495-4d70-99bf-b57764a2cff4-openshift-service-ca" pod "ee7795bd-0495-4d70-99bf-b57764a2cff4" (UID: "ee7795bd-0495-4d70-99bf-b57764a2cff4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:18:33.253634 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.253608    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee7795bd-0495-4d70-99bf-b57764a2cff4-kube-api-access-qfjd9" pod "ee7795bd-0495-4d70-99bf-b57764a2cff4" (UID: "ee7795bd-0495-4d70-99bf-b57764a2cff4"). InnerVolumeSpecName "kube-api-access-qfjd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:33.254188 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.254166    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-metrics-certs" pod "ee7795bd-0495-4d70-99bf-b57764a2cff4" (UID: "ee7795bd-0495-4d70-99bf-b57764a2cff4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:18:33.254307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.254198    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls" pod "ee7795bd-0495-4d70-99bf-b57764a2cff4" (UID: "ee7795bd-0495-4d70-99bf-b57764a2cff4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:18:33.303461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.303425    6000 generic.go:356] "Generic (PLEG): container finished" podID="ee7795bd-0495-4d70-99bf-b57764a2cff4" containerID="755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006" exitCode=0
Nov 03 01:18:33.303639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.303488    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"
Nov 03 01:18:33.303639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.303490    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd" event={"ID":"ee7795bd-0495-4d70-99bf-b57764a2cff4","Type":"ContainerDied","Data":"755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006"}
Nov 03 01:18:33.303639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.303559    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd" event={"ID":"ee7795bd-0495-4d70-99bf-b57764a2cff4","Type":"ContainerDied","Data":"d8d057e4c6fc8231383bcae318c693075f7b4c3cbad50ed86767d503e569e57c"}
Nov 03 01:18:33.303639 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.303582    6000 scope.go:122] "RemoveContainer" containerID="755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006"
Nov 03 01:18:33.313307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.313249    6000 scope.go:122] "RemoveContainer" containerID="755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006"
Nov 03 01:18:33.313643 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:33.313617    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006\": container with ID starting with 755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006 not found: ID does not exist" containerID="755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006"
Nov 03 01:18:33.313762 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.313653    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006"} err="failed to get container status \"755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006\": rpc error: code = NotFound desc = could not find container \"755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006\": container with ID starting with 755c9e2be6bd9b4efbb2c4d8d0ca4b2cb1c961945d0e1492b4335a8698e92006 not found: ID does not exist"
Nov 03 01:18:33.352112 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.352079    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"]
Nov 03 01:18:33.352613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.352588    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee7795bd-0495-4d70-99bf-b57764a2cff4-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:33.352720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.352617    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:33.352720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.352631    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfjd9\" (UniqueName: \"kubernetes.io/projected/ee7795bd-0495-4d70-99bf-b57764a2cff4-kube-api-access-qfjd9\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:33.352720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.352643    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ee7795bd-0495-4d70-99bf-b57764a2cff4-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:33.353896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:33.353875    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-0e3bc4e3-556d88b87d-xzkfd"]
Nov 03 01:18:34.565881 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:34.565848    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee7795bd-0495-4d70-99bf-b57764a2cff4" path="/var/lib/kubelet/pods/ee7795bd-0495-4d70-99bf-b57764a2cff4/volumes"
Nov 03 01:18:34.920539 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:34.920504    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:18:35.127862 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.127818    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r" podUID="654a7131-ef7e-4ef2-96ac-985f7b89b19f" containerName="payload-processing" containerID="cri-o://4c3ad4e49f4562f6d2160a165687112de1eba0117d238cdc94fc410f0e2f1fd8" gracePeriod=25
Nov 03 01:18:35.160409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.160372    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942" podUID="75e465bd-eed8-4e0b-b2cf-5fad31038064" containerName="payload-pre-processing" containerID="cri-o://7cd8208476d56dc302cb6bf355d664369fda23f9d89e99412cb40171da4d4924" gracePeriod=25
Nov 03 01:18:35.317981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.317948    6000 generic.go:356] "Generic (PLEG): container finished" podID="75e465bd-eed8-4e0b-b2cf-5fad31038064" containerID="7cd8208476d56dc302cb6bf355d664369fda23f9d89e99412cb40171da4d4924" exitCode=0
Nov 03 01:18:35.318176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.318077    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942" event={"ID":"75e465bd-eed8-4e0b-b2cf-5fad31038064","Type":"ContainerDied","Data":"7cd8208476d56dc302cb6bf355d664369fda23f9d89e99412cb40171da4d4924"}
Nov 03 01:18:35.319824 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.319792    6000 generic.go:356] "Generic (PLEG): container finished" podID="654a7131-ef7e-4ef2-96ac-985f7b89b19f" containerID="4c3ad4e49f4562f6d2160a165687112de1eba0117d238cdc94fc410f0e2f1fd8" exitCode=0
Nov 03 01:18:35.319948 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.319823    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r" event={"ID":"654a7131-ef7e-4ef2-96ac-985f7b89b19f","Type":"ContainerDied","Data":"4c3ad4e49f4562f6d2160a165687112de1eba0117d238cdc94fc410f0e2f1fd8"}
Nov 03 01:18:35.405608 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.405586    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:35.433341 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.433318    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:35.577660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.577574    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/654a7131-ef7e-4ef2-96ac-985f7b89b19f-kube-api-access-zdkxn\" (UniqueName: \"kubernetes.io/projected/654a7131-ef7e-4ef2-96ac-985f7b89b19f-kube-api-access-zdkxn\") pod \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") "
Nov 03 01:18:35.577660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.577611    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/654a7131-ef7e-4ef2-96ac-985f7b89b19f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a7131-ef7e-4ef2-96ac-985f7b89b19f-plugins-config-volume\") pod \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\" (UID: \"654a7131-ef7e-4ef2-96ac-985f7b89b19f\") "
Nov 03 01:18:35.577660 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.577661    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75e465bd-eed8-4e0b-b2cf-5fad31038064-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e465bd-eed8-4e0b-b2cf-5fad31038064-plugins-config-volume\") pod \"75e465bd-eed8-4e0b-b2cf-5fad31038064\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") "
Nov 03 01:18:35.578231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.577693    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75e465bd-eed8-4e0b-b2cf-5fad31038064-kube-api-access-hk8bn\" (UniqueName: \"kubernetes.io/projected/75e465bd-eed8-4e0b-b2cf-5fad31038064-kube-api-access-hk8bn\") pod \"75e465bd-eed8-4e0b-b2cf-5fad31038064\" (UID: \"75e465bd-eed8-4e0b-b2cf-5fad31038064\") "
Nov 03 01:18:35.578231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.578037    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e465bd-eed8-4e0b-b2cf-5fad31038064-plugins-config-volume" pod "75e465bd-eed8-4e0b-b2cf-5fad31038064" (UID: "75e465bd-eed8-4e0b-b2cf-5fad31038064"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:18:35.578231 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.578080    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/654a7131-ef7e-4ef2-96ac-985f7b89b19f-plugins-config-volume" pod "654a7131-ef7e-4ef2-96ac-985f7b89b19f" (UID: "654a7131-ef7e-4ef2-96ac-985f7b89b19f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:18:35.580067 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.580036    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/654a7131-ef7e-4ef2-96ac-985f7b89b19f-kube-api-access-zdkxn" pod "654a7131-ef7e-4ef2-96ac-985f7b89b19f" (UID: "654a7131-ef7e-4ef2-96ac-985f7b89b19f"). InnerVolumeSpecName "kube-api-access-zdkxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:35.580207 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.580114    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75e465bd-eed8-4e0b-b2cf-5fad31038064-kube-api-access-hk8bn" pod "75e465bd-eed8-4e0b-b2cf-5fad31038064" (UID: "75e465bd-eed8-4e0b-b2cf-5fad31038064"). InnerVolumeSpecName "kube-api-access-hk8bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:35.679710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.679669    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdkxn\" (UniqueName: \"kubernetes.io/projected/654a7131-ef7e-4ef2-96ac-985f7b89b19f-kube-api-access-zdkxn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:35.679710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.679712    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/654a7131-ef7e-4ef2-96ac-985f7b89b19f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:35.679945 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.679727    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75e465bd-eed8-4e0b-b2cf-5fad31038064-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:35.679945 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.679740    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk8bn\" (UniqueName: \"kubernetes.io/projected/75e465bd-eed8-4e0b-b2cf-5fad31038064-kube-api-access-hk8bn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:35.970398 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.970360    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"]
Nov 03 01:18:35.970683 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:35.970654    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" containerID="cri-o://8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33" gracePeriod=30
Nov 03 01:18:36.325951 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.325862    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"
Nov 03 01:18:36.325951 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.325879    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r" event={"ID":"654a7131-ef7e-4ef2-96ac-985f7b89b19f","Type":"ContainerDied","Data":"e9a204b2e56e75d03b3d00ca409f6c953c5f932ebc312f7e68fe8e488c327777"}
Nov 03 01:18:36.326213 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.325956    6000 scope.go:122] "RemoveContainer" containerID="4c3ad4e49f4562f6d2160a165687112de1eba0117d238cdc94fc410f0e2f1fd8"
Nov 03 01:18:36.327298 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.327231    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942" event={"ID":"75e465bd-eed8-4e0b-b2cf-5fad31038064","Type":"ContainerDied","Data":"21f1e5ea8f5bcc668612a9dcd52f9c62ccc18d43f766d3561b782f2dd0b3d220"}
Nov 03 01:18:36.327378 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.327300    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"
Nov 03 01:18:36.338705 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.338682    6000 scope.go:122] "RemoveContainer" containerID="7cd8208476d56dc302cb6bf355d664369fda23f9d89e99412cb40171da4d4924"
Nov 03 01:18:36.386835 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.386805    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"]
Nov 03 01:18:36.388426 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.388406    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3bc4e3-5dd474dd85-bqq7r"]
Nov 03 01:18:36.419731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.419701    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"]
Nov 03 01:18:36.421654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.421634    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3bc4e3-69bd6574d7-f9942"]
Nov 03 01:18:36.565383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.565346    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="654a7131-ef7e-4ef2-96ac-985f7b89b19f" path="/var/lib/kubelet/pods/654a7131-ef7e-4ef2-96ac-985f7b89b19f/volumes"
Nov 03 01:18:36.565811 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:36.565796    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75e465bd-eed8-4e0b-b2cf-5fad31038064" path="/var/lib/kubelet/pods/75e465bd-eed8-4e0b-b2cf-5fad31038064/volumes"
Nov 03 01:18:39.142347 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:39.142314    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:18:39.799673 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:39.799635    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-9crlh"]
Nov 03 01:18:39.799884 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:39.799858    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerName="manager" containerID="cri-o://d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab" gracePeriod=30
Nov 03 01:18:40.043971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.043947    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 01:18:40.223370 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.223337    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-kube-api-access-56p25\" (UniqueName: \"kubernetes.io/projected/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-kube-api-access-56p25\") pod \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") "
Nov 03 01:18:40.223862 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.223431    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-webhook-cert\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-webhook-cert\") pod \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") "
Nov 03 01:18:40.223862 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.223489    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-metrics-certs\") pod \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\" (UID: \"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5\") "
Nov 03 01:18:40.225681 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.225651    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-metrics-certs" pod "65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" (UID: "65984e3f-0d1a-4d8c-ac3a-ca76eda948a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:18:40.225784 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.225675    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-webhook-cert" pod "65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" (UID: "65984e3f-0d1a-4d8c-ac3a-ca76eda948a5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:18:40.225784 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.225719    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-kube-api-access-56p25" pod "65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" (UID: "65984e3f-0d1a-4d8c-ac3a-ca76eda948a5"). InnerVolumeSpecName "kube-api-access-56p25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:40.324940 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.324902    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56p25\" (UniqueName: \"kubernetes.io/projected/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-kube-api-access-56p25\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:40.324940 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.324936    6000 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-webhook-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:40.324940 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.324945    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:40.349187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.349156    6000 generic.go:356] "Generic (PLEG): container finished" podID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerID="d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab" exitCode=0
Nov 03 01:18:40.349366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.349208    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh"
Nov 03 01:18:40.349366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.349226    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" event={"ID":"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5","Type":"ContainerDied","Data":"d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab"}
Nov 03 01:18:40.349366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.349283    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-9crlh" event={"ID":"65984e3f-0d1a-4d8c-ac3a-ca76eda948a5","Type":"ContainerDied","Data":"d1cddab9f0a0cba5ae1dd784f40f7a5965484e350d884c8b2636813c68108dd4"}
Nov 03 01:18:40.349366 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.349306    6000 scope.go:122] "RemoveContainer" containerID="d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab"
Nov 03 01:18:40.362336 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.362318    6000 scope.go:122] "RemoveContainer" containerID="4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec"
Nov 03 01:18:40.370238 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.370219    6000 scope.go:122] "RemoveContainer" containerID="d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab"
Nov 03 01:18:40.370516 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:40.370494    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab\": container with ID starting with d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab not found: ID does not exist" containerID="d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab"
Nov 03 01:18:40.370609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.370528    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab"} err="failed to get container status \"d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab\": rpc error: code = NotFound desc = could not find container \"d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab\": container with ID starting with d9a20ddb06af1874cf1565f91a3dbb863d70e1dd3796ce40ccf9b91dce0e10ab not found: ID does not exist"
Nov 03 01:18:40.370609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.370554    6000 scope.go:122] "RemoveContainer" containerID="4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec"
Nov 03 01:18:40.370808 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:40.370790    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec\": container with ID starting with 4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec not found: ID does not exist" containerID="4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec"
Nov 03 01:18:40.370856 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.370814    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec"} err="failed to get container status \"4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec\": rpc error: code = NotFound desc = could not find container \"4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec\": container with ID starting with 4e2f7c10cc882944f7838b5f35b3866939eb3cd8fe033b9f561ce1e2ec8b52ec not found: ID does not exist"
Nov 03 01:18:40.396322 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.396293    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-9crlh"]
Nov 03 01:18:40.407820 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.407790    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-9crlh"]
Nov 03 01:18:40.565490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:40.565399    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" path="/var/lib/kubelet/pods/65984e3f-0d1a-4d8c-ac3a-ca76eda948a5/volumes"
Nov 03 01:18:41.074207 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.074164    6000 patch_prober.go:29] interesting pod/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused" start-of-body=
Nov 03 01:18:41.074410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.074225    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.48:15021/healthz/ready\": dial tcp 10.134.1.48:15021: connect: connection refused"
Nov 03 01:18:41.219326 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.219300    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:41.332820 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.332744    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istiod-ca-cert\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.332820 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.332789    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-socket\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.332820 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.332817    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-podinfo\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.332865    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-data\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.332919    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-kube-api-access-f78ss\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-kube-api-access-f78ss\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.332975    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-credential-socket\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.332993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-token\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-token\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333016    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-certs\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333038    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-socket" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333148    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istiod-ca-cert" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333191    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-data" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333199    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-envoy\") pod \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\" (UID: \"069b60ab-cde7-4e2a-adf1-b391d1e9793e\") "
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333333    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-credential-socket" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:41.333474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333347    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-certs" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:41.334248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333592    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.334248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333609    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.334248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333624    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.334248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333636    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.334248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.333644    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.335497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.335469    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-podinfo" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:18:41.335635 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.335614    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-kube-api-access-f78ss" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "kube-api-access-f78ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:41.335829 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.335809    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-token" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:18:41.335899 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.335824    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-envoy" pod "069b60ab-cde7-4e2a-adf1-b391d1e9793e" (UID: "069b60ab-cde7-4e2a-adf1-b391d1e9793e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:18:41.356163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.356132    6000 generic.go:356] "Generic (PLEG): container finished" podID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerID="8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33" exitCode=0
Nov 03 01:18:41.356302 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.356179    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"
Nov 03 01:18:41.356302 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.356189    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" event={"ID":"069b60ab-cde7-4e2a-adf1-b391d1e9793e","Type":"ContainerDied","Data":"8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33"}
Nov 03 01:18:41.356453 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.356305    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2" event={"ID":"069b60ab-cde7-4e2a-adf1-b391d1e9793e","Type":"ContainerDied","Data":"5c5dc2e6a5e5e3f221233eafd5785a6d59c73f555e7e57d3d9a22c509cdc6138"}
Nov 03 01:18:41.356453 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.356341    6000 scope.go:122] "RemoveContainer" containerID="8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33"
Nov 03 01:18:41.366171 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.366150    6000 scope.go:122] "RemoveContainer" containerID="fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a"
Nov 03 01:18:41.373970 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.373948    6000 scope.go:122] "RemoveContainer" containerID="8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33"
Nov 03 01:18:41.374221 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:41.374202    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33\": container with ID starting with 8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33 not found: ID does not exist" containerID="8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33"
Nov 03 01:18:41.374321 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.374233    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33"} err="failed to get container status \"8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33\": rpc error: code = NotFound desc = could not find container \"8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33\": container with ID starting with 8ba6faf5ff44db006ffb5e22941c2b1a082f949d56a23b67492bc35e3cac1f33 not found: ID does not exist"
Nov 03 01:18:41.374321 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.374272    6000 scope.go:122] "RemoveContainer" containerID="fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a"
Nov 03 01:18:41.374553 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:41.374535    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a\": container with ID starting with fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a not found: ID does not exist" containerID="fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a"
Nov 03 01:18:41.374594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.374560    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a"} err="failed to get container status \"fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a\": rpc error: code = NotFound desc = could not find container \"fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a\": container with ID starting with fc3bff57137f6cfff537655c1b6ed07c468ef061386d5daeb10eaf4d2190d36a not found: ID does not exist"
Nov 03 01:18:41.404176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.404149    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"]
Nov 03 01:18:41.407684 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.407662    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-0e3bc4e3-openshift-default-6f994578c9-5rmt2"]
Nov 03 01:18:41.434817 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.434796    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f78ss\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-kube-api-access-f78ss\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.434888 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.434817    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.434888 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.434828    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:41.434888 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:41.434837    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/069b60ab-cde7-4e2a-adf1-b391d1e9793e-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:18:42.446080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446046    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-b9ncb"]
Nov 03 01:18:42.446509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446457    6000 cpu_manager.go:414] "containerMap: removing container" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy"
Nov 03 01:18:42.446509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446467    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy"
Nov 03 01:18:42.446509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446478    6000 cpu_manager.go:414] "containerMap: removing container" podUID="75e465bd-eed8-4e0b-b2cf-5fad31038064" containerName="payload-pre-processing"
Nov 03 01:18:42.446509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446484    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75e465bd-eed8-4e0b-b2cf-5fad31038064" containerName="payload-pre-processing"
Nov 03 01:18:42.446509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446499    6000 cpu_manager.go:414] "containerMap: removing container" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerName="manager"
Nov 03 01:18:42.446509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446503    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerName="manager"
Nov 03 01:18:42.446509 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446513    6000 cpu_manager.go:414] "containerMap: removing container" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerName="manager"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446518    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerName="manager"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446529    6000 cpu_manager.go:414] "containerMap: removing container" podUID="654a7131-ef7e-4ef2-96ac-985f7b89b19f" containerName="payload-processing"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446534    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="654a7131-ef7e-4ef2-96ac-985f7b89b19f" containerName="payload-processing"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446545    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ee7795bd-0495-4d70-99bf-b57764a2cff4" containerName="maas-api"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446549    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee7795bd-0495-4d70-99bf-b57764a2cff4" containerName="maas-api"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446605    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446611    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="75e465bd-eed8-4e0b-b2cf-5fad31038064" containerName="payload-pre-processing"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446618    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerName="manager"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446625    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="65984e3f-0d1a-4d8c-ac3a-ca76eda948a5" containerName="manager"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446633    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="654a7131-ef7e-4ef2-96ac-985f7b89b19f" containerName="payload-processing"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446639    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee7795bd-0495-4d70-99bf-b57764a2cff4" containerName="maas-api"
Nov 03 01:18:42.446736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.446646    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy"
Nov 03 01:18:42.450052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.450033    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.464166 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.464141    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 01:18:42.475500 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.475469    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 01:18:42.475645 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.475596    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 01:18:42.475724 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.475682    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sfqns\""
Nov 03 01:18:42.477101 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.477078    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-b9ncb"]
Nov 03 01:18:42.565633 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.565604    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" path="/var/lib/kubelet/pods/069b60ab-cde7-4e2a-adf1-b391d1e9793e/volumes"
Nov 03 01:18:42.647129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.647099    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-webhook-cert\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.647129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.647130    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xwwl\" (UniqueName: \"kubernetes.io/projected/55e79348-30db-4697-a516-a2a0c0b2ca03-kube-api-access-9xwwl\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.647387 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.647212    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-metrics-certs\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.748050 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.747969    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-webhook-cert\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.748050 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.748002    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xwwl\" (UniqueName: \"kubernetes.io/projected/55e79348-30db-4697-a516-a2a0c0b2ca03-kube-api-access-9xwwl\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.748050 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.748040    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-metrics-certs\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.750625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.750598    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-metrics-certs\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.750778 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.750755    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-webhook-cert\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:42.766499 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:42.766475    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xwwl\" (UniqueName: \"kubernetes.io/projected/55e79348-30db-4697-a516-a2a0c0b2ca03-kube-api-access-9xwwl\") pod \"maas-controller-bc8c7d7fb-b9ncb\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") " pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:43.062736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:43.062651    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:43.352557 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:43.352533    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-b9ncb"]
Nov 03 01:18:43.354551 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:18:43.354525    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55e79348_30db_4697_a516_a2a0c0b2ca03.slice/crio-ffc879e3d79146fbdc44391c968825f8b4387a308976c121eeae46afcebad0d5 WatchSource:0}: Error finding container ffc879e3d79146fbdc44391c968825f8b4387a308976c121eeae46afcebad0d5: Status 404 returned error can't find the container with id ffc879e3d79146fbdc44391c968825f8b4387a308976c121eeae46afcebad0d5
Nov 03 01:18:43.367519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:43.367487    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb" event={"ID":"55e79348-30db-4697-a516-a2a0c0b2ca03","Type":"ContainerStarted","Data":"ffc879e3d79146fbdc44391c968825f8b4387a308976c121eeae46afcebad0d5"}
Nov 03 01:18:44.373692 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:44.373659    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb" event={"ID":"55e79348-30db-4697-a516-a2a0c0b2ca03","Type":"ContainerStarted","Data":"d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3"}
Nov 03 01:18:44.374218 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:44.373760    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:44.414917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:44.414861    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb" podStartSLOduration=2.010542346 podStartE2EDuration="2.414846818s" totalImagesPullingTime="404.304472ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:44.413628673 +0000 UTC m=+4361.968980712" watchObservedRunningTime="2026-11-03 01:18:44.414846818 +0000 UTC m=+4361.970198835"
Nov 03 01:18:49.895409 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:18:49.895373    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:18:55.380943 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:55.380907    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:18:58.494500 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.494468    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"]
Nov 03 01:18:58.494938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.494911    6000 cpu_manager.go:414] "containerMap: removing container" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy"
Nov 03 01:18:58.494938 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.494924    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="069b60ab-cde7-4e2a-adf1-b391d1e9793e" containerName="istio-proxy"
Nov 03 01:18:58.497145 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.497120    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.506495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.506468    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"]
Nov 03 01:18:58.509371 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.509354    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.524558 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.524536    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"]
Nov 03 01:18:58.531067 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.531043    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"]
Nov 03 01:18:58.551049 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.551013    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-v9b8n\""
Nov 03 01:18:58.598495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.598469    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 03 01:18:58.602941 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.602915    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.602953    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.602974    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603145 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603047    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603145 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603119    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603145 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603143    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603163    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z78z\" (UniqueName: \"kubernetes.io/projected/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-kube-api-access-4z78z\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603181    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603304    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603325    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603348    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603373    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zpt4\" (UniqueName: \"kubernetes.io/projected/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-kube-api-access-9zpt4\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603445    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603475    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603499    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603559 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603521    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.603878 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603569    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.603878 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.603609    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.704896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.704855    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.704896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.704893    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.704916    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.704934    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.704954    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.704986    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705005    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705025    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705043    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705087    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705119    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705170 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705151    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z78z\" (UniqueName: \"kubernetes.io/projected/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-kube-api-access-4z78z\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705177    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705228    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705274    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705307    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705341    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zpt4\" (UniqueName: \"kubernetes.io/projected/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-kube-api-access-9zpt4\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705356    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705373    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705402    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705409    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705453    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705640    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.705869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.705812    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.706498 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.706099    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.706498 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.706105    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.706498 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.706373    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.706640 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.706557    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.708118 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.708095    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.708212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.708147    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.708368 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.708346    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.708612 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.708597    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.724061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.724039    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.724184 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.724100    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.724375 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.724350    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zpt4\" (UniqueName: \"kubernetes.io/projected/b869da23-2c4e-4e69-a1b7-a9ae8585d8a1-kube-api-access-9zpt4\") pod \"maas-default-gateway-openshift-default-6cc458d887-w99jj\" (UID: \"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:58.724510 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.724495    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z78z\" (UniqueName: \"kubernetes.io/projected/ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf-kube-api-access-4z78z\") pod \"maas-default-gateway-openshift-default-6cc458d887-t297d\" (UID: \"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.812121 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.812030    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:58.821075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:58.821056    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:18:59.128568 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.128534    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"]
Nov 03 01:18:59.129883 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:18:59.129850    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb869da23_2c4e_4e69_a1b7_a9ae8585d8a1.slice/crio-ae84b740e3cc12ebabc9d6fcdcc4ae78d707f6c3333b0705a1b9c2c48ce06fb2 WatchSource:0}: Error finding container ae84b740e3cc12ebabc9d6fcdcc4ae78d707f6c3333b0705a1b9c2c48ce06fb2: Status 404 returned error can't find the container with id ae84b740e3cc12ebabc9d6fcdcc4ae78d707f6c3333b0705a1b9c2c48ce06fb2
Nov 03 01:18:59.130070 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.130054    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"]
Nov 03 01:18:59.132074 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.132047    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:59.132139 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.132109    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:59.132177 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.132136    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:59.140773 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:18:59.140747    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7bba15_a9a5_4baa_9bc5_5383fcc64bdf.slice/crio-7f679388ba2dbd052f3b7dda42d1744b59350382a30e112a50ed4df5da63de19 WatchSource:0}: Error finding container 7f679388ba2dbd052f3b7dda42d1744b59350382a30e112a50ed4df5da63de19: Status 404 returned error can't find the container with id 7f679388ba2dbd052f3b7dda42d1744b59350382a30e112a50ed4df5da63de19
Nov 03 01:18:59.142807 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.142783    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:59.142904 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.142839    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:59.142904 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.142865    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:18:59.444493 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.444383    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" event={"ID":"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1","Type":"ContainerStarted","Data":"9025cfcbbda9bd0d6a7a12e00428a145642ca96d106eae3dc3ba4069b6196c75"}
Nov 03 01:18:59.444493 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.444485    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" event={"ID":"b869da23-2c4e-4e69-a1b7-a9ae8585d8a1","Type":"ContainerStarted","Data":"ae84b740e3cc12ebabc9d6fcdcc4ae78d707f6c3333b0705a1b9c2c48ce06fb2"}
Nov 03 01:18:59.446283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.446228    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" event={"ID":"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf","Type":"ContainerStarted","Data":"f302db1b180803507a670d62c8927389e2defb867fa39d974851b4dd7528ecdb"}
Nov 03 01:18:59.446433 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.446303    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" event={"ID":"ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf","Type":"ContainerStarted","Data":"7f679388ba2dbd052f3b7dda42d1744b59350382a30e112a50ed4df5da63de19"}
Nov 03 01:18:59.486466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.486382    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podStartSLOduration=1.4863628229999999 podStartE2EDuration="1.486362823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:59.483669751 +0000 UTC m=+4377.039021777" watchObservedRunningTime="2026-11-03 01:18:59.486362823 +0000 UTC m=+4377.041714840"
Nov 03 01:18:59.523042 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.522987    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podStartSLOduration=1.522972708 podStartE2EDuration="1.522972708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:18:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:18:59.522071916 +0000 UTC m=+4377.077423942" watchObservedRunningTime="2026-11-03 01:18:59.522972708 +0000 UTC m=+4377.078324731"
Nov 03 01:18:59.815562 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.815385    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:18:59.828282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:18:59.824374    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:19:00.819282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:00.816170    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Nov 03 01:19:00.819282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:00.816278    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Nov 03 01:19:00.833294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:00.829363    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body=
Nov 03 01:19:00.833294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:00.829423    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused"
Nov 03 01:19:01.818280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:01.814687    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Nov 03 01:19:01.818280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:01.814768    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Nov 03 01:19:01.824291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:01.823665    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body=
Nov 03 01:19:01.824291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:01.823715    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused"
Nov 03 01:19:02.311284 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:02.306696    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:19:02.815751 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:02.815702    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Nov 03 01:19:02.815985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:02.815773    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Nov 03 01:19:02.832304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:02.823656    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body=
Nov 03 01:19:02.832304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:02.823711    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused"
Nov 03 01:19:03.814280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:03.813729    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Nov 03 01:19:03.814280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:03.813811    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Nov 03 01:19:03.825280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:03.823688    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body=
Nov 03 01:19:03.825280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:03.823748    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused"
Nov 03 01:19:04.816282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:04.812663    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Nov 03 01:19:04.816282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:04.812728    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Nov 03 01:19:04.824280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:04.823649    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body=
Nov 03 01:19:04.824280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:04.823729    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused"
Nov 03 01:19:05.812872 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:05.812827    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body=
Nov 03 01:19:05.813075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:05.812902    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused"
Nov 03 01:19:05.822297 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:05.822244    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body=
Nov 03 01:19:05.822669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:05.822318    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused"
Nov 03 01:19:06.822379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:06.822330    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body=
Nov 03 01:19:06.822960 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:06.822404    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused"
Nov 03 01:19:07.812901 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:07.812855    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:19:07.813142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:07.812937    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:19:08.854512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:08.854458    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:19:08.854969 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:08.854536    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:19:08.863408 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:08.863380    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-w99jj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:19:08.863484 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:08.863428    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj" podUID="b869da23-2c4e-4e69-a1b7-a9ae8585d8a1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:19:09.849958 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:09.849911    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:19:09.850511 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:09.850479    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:19:09.851151 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:09.851131    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-w99jj"
Nov 03 01:19:09.896420 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:09.896374    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-t297d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:19:09.896896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:09.896430    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d" podUID="ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:19:10.302910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:10.302866    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:19:10.303299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:10.303283    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:19:10.304002 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:10.303983    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-t297d"
Nov 03 01:19:12.455853 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.455811    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-b9ncb"]
Nov 03 01:19:12.456391 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.455997    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb" podUID="55e79348-30db-4697-a516-a2a0c0b2ca03" containerName="manager" containerID="cri-o://d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3" gracePeriod=30
Nov 03 01:19:12.707182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.707120    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:19:12.781066 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.781028    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-metrics-certs\") pod \"55e79348-30db-4697-a516-a2a0c0b2ca03\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") "
Nov 03 01:19:12.781292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.781102    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55e79348-30db-4697-a516-a2a0c0b2ca03-kube-api-access-9xwwl\" (UniqueName: \"kubernetes.io/projected/55e79348-30db-4697-a516-a2a0c0b2ca03-kube-api-access-9xwwl\") pod \"55e79348-30db-4697-a516-a2a0c0b2ca03\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") "
Nov 03 01:19:12.781292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.781124    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-webhook-cert\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-webhook-cert\") pod \"55e79348-30db-4697-a516-a2a0c0b2ca03\" (UID: \"55e79348-30db-4697-a516-a2a0c0b2ca03\") "
Nov 03 01:19:12.783460 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.783428    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55e79348-30db-4697-a516-a2a0c0b2ca03-kube-api-access-9xwwl" pod "55e79348-30db-4697-a516-a2a0c0b2ca03" (UID: "55e79348-30db-4697-a516-a2a0c0b2ca03"). InnerVolumeSpecName "kube-api-access-9xwwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:19:12.783578 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.783457    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-metrics-certs" pod "55e79348-30db-4697-a516-a2a0c0b2ca03" (UID: "55e79348-30db-4697-a516-a2a0c0b2ca03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:12.783578 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.783469    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-webhook-cert" pod "55e79348-30db-4697-a516-a2a0c0b2ca03" (UID: "55e79348-30db-4697-a516-a2a0c0b2ca03"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:19:12.882707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.882668    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:19:12.882707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.882707    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xwwl\" (UniqueName: \"kubernetes.io/projected/55e79348-30db-4697-a516-a2a0c0b2ca03-kube-api-access-9xwwl\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:19:12.882976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:12.882719    6000 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55e79348-30db-4697-a516-a2a0c0b2ca03-webhook-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:19:13.530135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.530093    6000 generic.go:356] "Generic (PLEG): container finished" podID="55e79348-30db-4697-a516-a2a0c0b2ca03" containerID="d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3" exitCode=0
Nov 03 01:19:13.530669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.530145    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb"
Nov 03 01:19:13.530669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.530186    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb" event={"ID":"55e79348-30db-4697-a516-a2a0c0b2ca03","Type":"ContainerDied","Data":"d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3"}
Nov 03 01:19:13.530669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.530248    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-b9ncb" event={"ID":"55e79348-30db-4697-a516-a2a0c0b2ca03","Type":"ContainerDied","Data":"ffc879e3d79146fbdc44391c968825f8b4387a308976c121eeae46afcebad0d5"}
Nov 03 01:19:13.530669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.530299    6000 scope.go:122] "RemoveContainer" containerID="d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3"
Nov 03 01:19:13.533536 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.533514    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"]
Nov 03 01:19:13.533975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.533959    6000 cpu_manager.go:414] "containerMap: removing container" podUID="55e79348-30db-4697-a516-a2a0c0b2ca03" containerName="manager"
Nov 03 01:19:13.534077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.533978    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55e79348-30db-4697-a516-a2a0c0b2ca03" containerName="manager"
Nov 03 01:19:13.534077 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.534062    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="55e79348-30db-4697-a516-a2a0c0b2ca03" containerName="manager"
Nov 03 01:19:13.536352 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.536332    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.539768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.539749    6000 scope.go:122] "RemoveContainer" containerID="d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3"
Nov 03 01:19:13.540109 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:19:13.540086    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3\": container with ID starting with d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3 not found: ID does not exist" containerID="d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3"
Nov 03 01:19:13.540381 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.540132    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3"} err="failed to get container status \"d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3\": rpc error: code = NotFound desc = could not find container \"d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3\": container with ID starting with d6edd925c97bc0b74b004275cb7a6551418ccc64aec72e1aaefd4945c4d5c7b3 not found: ID does not exist"
Nov 03 01:19:13.562575 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.562548    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"]
Nov 03 01:19:13.565293 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.565075    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.571636 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.571613    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"]
Nov 03 01:19:13.602765 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.602731    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"]
Nov 03 01:19:13.640333 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.640301    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-b9ncb"]
Nov 03 01:19:13.648519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.648495    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-b9ncb"]
Nov 03 01:19:13.691944 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.691916    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692107 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.691950    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692107 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.691976    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzm47\" (UniqueName: \"kubernetes.io/projected/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-kube-api-access-gzm47\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692107 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692047    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692107 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692096    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692137    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692156    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692177    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692208    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692233    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692340    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65d6c3e-af31-47a7-a807-15531faca35f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692360    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692390    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692412    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692495    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.692625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692579    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wj2sw\" (UniqueName: \"kubernetes.io/projected/f65d6c3e-af31-47a7-a807-15531faca35f-kube-api-access-wj2sw\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692624    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65d6c3e-af31-47a7-a807-15531faca35f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.692845 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.692662    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65d6c3e-af31-47a7-a807-15531faca35f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.793865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793797    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzm47\" (UniqueName: \"kubernetes.io/projected/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-kube-api-access-gzm47\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.793865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793833    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.793865 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793857    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793875    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793904    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793937    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793959    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.793975    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794000    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65d6c3e-af31-47a7-a807-15531faca35f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794028    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.794128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794123    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794315    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794315    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794373    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794444    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794478    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794508    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wj2sw\" (UniqueName: \"kubernetes.io/projected/f65d6c3e-af31-47a7-a807-15531faca35f-kube-api-access-wj2sw\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794544    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65d6c3e-af31-47a7-a807-15531faca35f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.794600 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794584    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65d6c3e-af31-47a7-a807-15531faca35f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.795080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794645    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.795080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794675    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.795080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794732    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.795080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.794787    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-workload-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.795349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.795188    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-credential-socket\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.795349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.795200    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-workload-certs\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.795349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.795337    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.795523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.795495    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-istio-data\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.795662 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.795644    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f65d6c3e-af31-47a7-a807-15531faca35f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.797032 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.797006    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f65d6c3e-af31-47a7-a807-15531faca35f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.797206 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.797188    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f65d6c3e-af31-47a7-a807-15531faca35f-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.797326 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.797184    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-envoy\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.797811 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.797788    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.813052 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.813030    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzm47\" (UniqueName: \"kubernetes.io/projected/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-kube-api-access-gzm47\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.813545 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.813520    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f65d6c3e-af31-47a7-a807-15531faca35f-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.813802 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.813783    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wj2sw\" (UniqueName: \"kubernetes.io/projected/f65d6c3e-af31-47a7-a807-15531faca35f-kube-api-access-wj2sw\") pod \"maas-default-gateway-openshift-default-6cc458d887-msgxv\" (UID: \"f65d6c3e-af31-47a7-a807-15531faca35f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:13.814074 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.814058    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad946480-1aaa-4a6c-826c-a0db0c4a9ac9-istio-token\") pod \"maas-default-gateway-openshift-default-6cc458d887-dhv97\" (UID: \"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.851868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.851834    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:13.879294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:13.879247    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:14.155461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.155166    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"]
Nov 03 01:19:14.157153 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:19:14.157129    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad946480_1aaa_4a6c_826c_a0db0c4a9ac9.slice/crio-6af0d5da9b490cf700fcd7dcd11bb2c46342fb4a86b9fefc8fee3e274be48cdf WatchSource:0}: Error finding container 6af0d5da9b490cf700fcd7dcd11bb2c46342fb4a86b9fefc8fee3e274be48cdf: Status 404 returned error can't find the container with id 6af0d5da9b490cf700fcd7dcd11bb2c46342fb4a86b9fefc8fee3e274be48cdf
Nov 03 01:19:14.159433 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.159402    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:19:14.159512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.159465    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:19:14.159512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.159494    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:19:14.194074 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.193999    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"]
Nov 03 01:19:14.207474 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:19:14.207448    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf65d6c3e_af31_47a7_a807_15531faca35f.slice/crio-ce1bcba6e577f252d6a5dea67fa7899dec2121577c157613d9acf9c22c90a185 WatchSource:0}: Error finding container ce1bcba6e577f252d6a5dea67fa7899dec2121577c157613d9acf9c22c90a185: Status 404 returned error can't find the container with id ce1bcba6e577f252d6a5dea67fa7899dec2121577c157613d9acf9c22c90a185
Nov 03 01:19:14.209686 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.209654    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:19:14.209841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.209719    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:19:14.209841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.209748    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:19:14.539760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.539669    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" event={"ID":"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9","Type":"ContainerStarted","Data":"91ab0d586be02384cb0651592aa2a2e6589f05f20d1c086e7225ff17f28b3509"}
Nov 03 01:19:14.539760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.539732    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" event={"ID":"ad946480-1aaa-4a6c-826c-a0db0c4a9ac9","Type":"ContainerStarted","Data":"6af0d5da9b490cf700fcd7dcd11bb2c46342fb4a86b9fefc8fee3e274be48cdf"}
Nov 03 01:19:14.541737 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.541709    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" event={"ID":"f65d6c3e-af31-47a7-a807-15531faca35f","Type":"ContainerStarted","Data":"672cd33d8a4e4f3ab33974b776411421b1767c012383eb7d1af64454f18dcf0d"}
Nov 03 01:19:14.541878 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.541752    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" event={"ID":"f65d6c3e-af31-47a7-a807-15531faca35f","Type":"ContainerStarted","Data":"ce1bcba6e577f252d6a5dea67fa7899dec2121577c157613d9acf9c22c90a185"}
Nov 03 01:19:14.564910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.564876    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55e79348-30db-4697-a516-a2a0c0b2ca03" path="/var/lib/kubelet/pods/55e79348-30db-4697-a516-a2a0c0b2ca03/volumes"
Nov 03 01:19:14.633054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.630708    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podStartSLOduration=1.6306767039999999 podStartE2EDuration="1.630676704s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:14.630209996 +0000 UTC m=+4392.185562081" watchObservedRunningTime="2026-11-03 01:19:14.630676704 +0000 UTC m=+4392.186028730"
Nov 03 01:19:14.633054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.631626    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podStartSLOduration=1.631615483 podStartE2EDuration="1.631615483s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:14.589893328 +0000 UTC m=+4392.145245365" watchObservedRunningTime="2026-11-03 01:19:14.631615483 +0000 UTC m=+4392.186967510"
Nov 03 01:19:14.852392 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.852364    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:14.879545 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.879517    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:14.909933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.906734    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-gcc5b"]
Nov 03 01:19:14.927280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.920567    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:14.931599 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.931567    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-gcc5b"]
Nov 03 01:19:14.939283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.938272    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Nov 03 01:19:14.939283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.938482    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Nov 03 01:19:14.939283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.939128    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-sfqns\""
Nov 03 01:19:14.939283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:14.939223    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Nov 03 01:19:15.022316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.016824    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4146cae1-50a3-4d36-b7af-65bb9c50655d-webhook-cert\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.022316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.016946    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4146cae1-50a3-4d36-b7af-65bb9c50655d-metrics-certs\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.022316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.017058    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trgl2\" (UniqueName: \"kubernetes.io/projected/4146cae1-50a3-4d36-b7af-65bb9c50655d-kube-api-access-trgl2\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.122329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.117876    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trgl2\" (UniqueName: \"kubernetes.io/projected/4146cae1-50a3-4d36-b7af-65bb9c50655d-kube-api-access-trgl2\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.122329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.117963    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4146cae1-50a3-4d36-b7af-65bb9c50655d-webhook-cert\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.122329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.118066    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4146cae1-50a3-4d36-b7af-65bb9c50655d-metrics-certs\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.126068 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.126038    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4146cae1-50a3-4d36-b7af-65bb9c50655d-webhook-cert\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.141303 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.139396    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4146cae1-50a3-4d36-b7af-65bb9c50655d-metrics-certs\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.142117 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.142092    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trgl2\" (UniqueName: \"kubernetes.io/projected/4146cae1-50a3-4d36-b7af-65bb9c50655d-kube-api-access-trgl2\") pod \"maas-controller-bc8c7d7fb-gcc5b\" (UID: \"4146cae1-50a3-4d36-b7af-65bb9c50655d\") " pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.249932 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.249886    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:15.610303 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.608988    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-bc8c7d7fb-gcc5b"]
Nov 03 01:19:15.853710 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.853670    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:19:15.853890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.853729    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused"
Nov 03 01:19:15.881019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.880936    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 03 01:19:15.881019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:15.880991    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 03 01:19:16.552295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:16.552183    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b" event={"ID":"4146cae1-50a3-4d36-b7af-65bb9c50655d","Type":"ContainerStarted","Data":"f7de82732680a3ed82f1e3e639634f6cc1ed9c629cfaf28a9bcdf6a97df67463"}
Nov 03 01:19:16.552295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:16.552244    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b" event={"ID":"4146cae1-50a3-4d36-b7af-65bb9c50655d","Type":"ContainerStarted","Data":"0492cf166c16f6851482bdcfe39351534237231630bc7c101493dd0b0121071f"}
Nov 03 01:19:16.852305 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:16.852243    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:19:16.852757 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:16.852361    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused"
Nov 03 01:19:16.879751 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:16.879721    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 03 01:19:16.879926 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:16.879775    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 03 01:19:17.556702 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:17.556671    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:17.852550 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:17.852513    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:19:17.852954 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:17.852577    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused"
Nov 03 01:19:17.879949 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:17.879924    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 03 01:19:17.880059 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:17.879963    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 03 01:19:18.852882 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:18.852841    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:19:18.853360 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:18.852899    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused"
Nov 03 01:19:18.879770 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:18.879741    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 03 01:19:18.879884 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:18.879785    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 03 01:19:19.852685 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:19.852647    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:19:19.852881 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:19.852705    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused"
Nov 03 01:19:19.880147 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:19.880112    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 03 01:19:19.880552 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:19.880160    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 03 01:19:19.942389 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:19:19.942347    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:19:20.853085 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:20.853044    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused" start-of-body=
Nov 03 01:19:20.853304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:20.853114    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": dial tcp 10.134.1.55:15021: connect: connection refused"
Nov 03 01:19:20.880002 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:20.879969    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused" start-of-body=
Nov 03 01:19:20.880157 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:20.880030    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": dial tcp 10.134.1.56:15021: connect: connection refused"
Nov 03 01:19:22.568917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:22.568886    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:22.569458 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:22.569159    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:22.570020 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:22.570001    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv"
Nov 03 01:19:22.633065 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:22.633001    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b" podStartSLOduration=8.20463653 podStartE2EDuration="8.632982069s" totalImagesPullingTime="428.345539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:19:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:19:16.625514072 +0000 UTC m=+4394.180866096" watchObservedRunningTime="2026-11-03 01:19:22.632982069 +0000 UTC m=+4400.188334095"
Nov 03 01:19:22.649932 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:22.649903    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:22.650585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:22.650555    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:22.651479 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:22.651462    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97"
Nov 03 01:19:28.566503 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:28.566476    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-bc8c7d7fb-gcc5b"
Nov 03 01:19:36.712924 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:36.712887    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:19:49.896613 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:19:49.896517    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:19:52.519586 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:52.519549    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:19:57.724943 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:19:57.724900    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:20:03.876283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:03.854017    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:20:07.945349 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:07.945309    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:20:20.030885 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:20:20.030801    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:20:44.021892 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:44.021857    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:20:46.516913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.516880    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"]
Nov 03 01:20:46.520929 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.520902    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.548609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.548581    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-a77df7ec-kserve-self-signed-certs\""
Nov 03 01:20:46.550082 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.550053    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"]
Nov 03 01:20:46.698041 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.697999    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-model-cache\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.698420 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.698388    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqmqm\" (UniqueName: \"kubernetes.io/projected/99d2f743-ff01-4f3a-a4f0-d48467661ea3-kube-api-access-bqmqm\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.698561 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.698498    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tls-certs\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.698561 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.698544    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tmp-dir\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.698692 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.698567    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-home\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.698692 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.698611    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-dshm\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.799585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799497    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-model-cache\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.799585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799560    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqmqm\" (UniqueName: \"kubernetes.io/projected/99d2f743-ff01-4f3a-a4f0-d48467661ea3-kube-api-access-bqmqm\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.799826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799613    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tls-certs\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.799826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799651    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tmp-dir\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.799826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799671    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-home\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.799826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799707    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-dshm\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.800064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799899    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-model-cache\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.800064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.799996    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tmp-dir\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.800064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.800021    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-home\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.802116 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.802094    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-dshm\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.802327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.802311    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tls-certs\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.817328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.817305    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqmqm\" (UniqueName: \"kubernetes.io/projected/99d2f743-ff01-4f3a-a4f0-d48467661ea3-kube-api-access-bqmqm\") pod \"e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.835330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.835303    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:46.871040 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.871009    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"]
Nov 03 01:20:46.877064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.876520    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:46.905629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.905593    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-prio-pool-a77df7ec-epp-sa-dockercfg-ws8p8\""
Nov 03 01:20:46.906620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:46.906589    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"]
Nov 03 01:20:47.007118 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.007077    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-tls-certs\") pod \"e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:47.007317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.007152    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8q64x\" (UniqueName: \"kubernetes.io/projected/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-kube-api-access-8q64x\") pod \"e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:47.108666 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.108639    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-tls-certs\") pod \"e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:47.108841 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.108711    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8q64x\" (UniqueName: \"kubernetes.io/projected/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-kube-api-access-8q64x\") pod \"e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:47.111345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.111319    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-tls-certs\") pod \"e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:47.126701 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.126676    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8q64x\" (UniqueName: \"kubernetes.io/projected/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-kube-api-access-8q64x\") pod \"e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") " pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:47.132158 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.132137    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"]
Nov 03 01:20:47.134134 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:20:47.134109    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99d2f743_ff01_4f3a_a4f0_d48467661ea3.slice/crio-6b3dd3c465861fc0c0f1a182a067f84abc6c5c55284e2673654cc34860685164 WatchSource:0}: Error finding container 6b3dd3c465861fc0c0f1a182a067f84abc6c5c55284e2673654cc34860685164: Status 404 returned error can't find the container with id 6b3dd3c465861fc0c0f1a182a067f84abc6c5c55284e2673654cc34860685164
Nov 03 01:20:47.203624 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.203587    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh" event={"ID":"99d2f743-ff01-4f3a-a4f0-d48467661ea3","Type":"ContainerStarted","Data":"6b3dd3c465861fc0c0f1a182a067f84abc6c5c55284e2673654cc34860685164"}
Nov 03 01:20:47.215921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.215882    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:47.513108 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:47.513074    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"]
Nov 03 01:20:47.514001 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:20:47.513978    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71fd5526_2d22_4aaf_88ab_649b4e3c9c80.slice/crio-2503bd2850fd38a453f1c72b89900cfda2e2dcd912c8fc3097e38112cb450b1d WatchSource:0}: Error finding container 2503bd2850fd38a453f1c72b89900cfda2e2dcd912c8fc3097e38112cb450b1d: Status 404 returned error can't find the container with id 2503bd2850fd38a453f1c72b89900cfda2e2dcd912c8fc3097e38112cb450b1d
Nov 03 01:20:48.209995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:48.209959    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh" event={"ID":"99d2f743-ff01-4f3a-a4f0-d48467661ea3","Type":"ContainerStarted","Data":"064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593"}
Nov 03 01:20:48.211241 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:48.211214    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" event={"ID":"71fd5526-2d22-4aaf-88ab-649b4e3c9c80","Type":"ContainerStarted","Data":"2503bd2850fd38a453f1c72b89900cfda2e2dcd912c8fc3097e38112cb450b1d"}
Nov 03 01:20:48.245919 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:48.245859    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh" podStartSLOduration=2.245842103 podStartE2EDuration="2.245842103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:48.245508573 +0000 UTC m=+4485.800860597" watchObservedRunningTime="2026-11-03 01:20:48.245842103 +0000 UTC m=+4485.801194128"
Nov 03 01:20:48.836345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:48.836312    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:49.218328 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:49.218289    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" event={"ID":"71fd5526-2d22-4aaf-88ab-649b4e3c9c80","Type":"ContainerStarted","Data":"5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c"}
Nov 03 01:20:49.255441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:49.255383    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" podStartSLOduration=1.646217179 podStartE2EDuration="3.255369592s" totalImagesPullingTime="1.609152413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:20:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:20:49.254308068 +0000 UTC m=+4486.809660086" watchObservedRunningTime="2026-11-03 01:20:49.255369592 +0000 UTC m=+4486.810721615"
Nov 03 01:20:49.895341 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:20:49.895303    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:20:49.943071 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:49.943040    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:20:50.223151 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:50.223064    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:20:53.652327 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:53.652283    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:20:53.652963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:53.652929    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:20:56.835572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:56.835528    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:56.848387 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:56.848360    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:20:57.291210 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:20:57.291179    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:21:19.895711 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:21:19.895624    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:21:21.231534 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:21.231505    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:21:30.044834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:30.044797    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:21:49.114393 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:49.114357    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:21:49.467734 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:49.467599    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"]
Nov 03 01:21:49.478048 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:49.478007    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"]
Nov 03 01:21:49.895630 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:21:49.895589    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:21:51.231617 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:51.231573    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:21:53.675583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:53.675530    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:21:53.676356 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:53.676302    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:21:53.691464 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:53.691429    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:21:53.691750 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:53.691722    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:21:54.430562 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:54.430523    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:21:59.625154 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:21:59.625120    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:22:01.231452 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:01.231416    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:22:04.494915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.494858    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh" podUID="99d2f743-ff01-4f3a-a4f0-d48467661ea3" containerName="main" containerID="cri-o://064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593" gracePeriod=45
Nov 03 01:22:04.500140 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.500086    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main" containerID="cri-o://5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c" gracePeriod=45
Nov 03 01:22:04.785903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.785863    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:22:04.808719 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.808689    6000 generic.go:356] "Generic (PLEG): container finished" podID="99d2f743-ff01-4f3a-a4f0-d48467661ea3" containerID="064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593" exitCode=0
Nov 03 01:22:04.808910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.808734    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"
Nov 03 01:22:04.808910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.808735    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh" event={"ID":"99d2f743-ff01-4f3a-a4f0-d48467661ea3","Type":"ContainerDied","Data":"064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593"}
Nov 03 01:22:04.808910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.808858    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh" event={"ID":"99d2f743-ff01-4f3a-a4f0-d48467661ea3","Type":"ContainerDied","Data":"6b3dd3c465861fc0c0f1a182a067f84abc6c5c55284e2673654cc34860685164"}
Nov 03 01:22:04.808910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.808881    6000 scope.go:122] "RemoveContainer" containerID="064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593"
Nov 03 01:22:04.817595 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.817574    6000 scope.go:122] "RemoveContainer" containerID="064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593"
Nov 03 01:22:04.817848 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:22:04.817827    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593\": container with ID starting with 064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593 not found: ID does not exist" containerID="064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593"
Nov 03 01:22:04.817894 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.817856    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593"} err="failed to get container status \"064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593\": rpc error: code = NotFound desc = could not find container \"064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593\": container with ID starting with 064d3a9e83b66d9a0c2bfa86c3de2e6ba494a3265b9e49b6fab6539f8456a593 not found: ID does not exist"
Nov 03 01:22:04.916983 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.916950    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-dshm\") pod \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") "
Nov 03 01:22:04.916983 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.916989    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tls-certs\" (UniqueName: \"kubernetes.io/secret/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tls-certs\") pod \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") "
Nov 03 01:22:04.917234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.917029    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-home\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-home\") pod \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") "
Nov 03 01:22:04.917234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.917084    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d2f743-ff01-4f3a-a4f0-d48467661ea3-kube-api-access-bqmqm\" (UniqueName: \"kubernetes.io/projected/99d2f743-ff01-4f3a-a4f0-d48467661ea3-kube-api-access-bqmqm\") pod \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") "
Nov 03 01:22:04.917234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.917211    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tmp-dir\") pod \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") "
Nov 03 01:22:04.917466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.917247    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-model-cache\") pod \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\" (UID: \"99d2f743-ff01-4f3a-a4f0-d48467661ea3\") "
Nov 03 01:22:04.917466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.917355    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-home" pod "99d2f743-ff01-4f3a-a4f0-d48467661ea3" (UID: "99d2f743-ff01-4f3a-a4f0-d48467661ea3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:22:04.917583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.917512    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-model-cache" pod "99d2f743-ff01-4f3a-a4f0-d48467661ea3" (UID: "99d2f743-ff01-4f3a-a4f0-d48467661ea3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:22:04.917583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.917534    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tmp-dir" pod "99d2f743-ff01-4f3a-a4f0-d48467661ea3" (UID: "99d2f743-ff01-4f3a-a4f0-d48467661ea3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:22:04.918232 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.918206    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:04.918330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.918239    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:04.918330 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.918274    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:04.919547 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.919524    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tls-certs" pod "99d2f743-ff01-4f3a-a4f0-d48467661ea3" (UID: "99d2f743-ff01-4f3a-a4f0-d48467661ea3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:22:04.919663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.919589    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-dshm" pod "99d2f743-ff01-4f3a-a4f0-d48467661ea3" (UID: "99d2f743-ff01-4f3a-a4f0-d48467661ea3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:22:04.919663 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:04.919611    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d2f743-ff01-4f3a-a4f0-d48467661ea3-kube-api-access-bqmqm" pod "99d2f743-ff01-4f3a-a4f0-d48467661ea3" (UID: "99d2f743-ff01-4f3a-a4f0-d48467661ea3"). InnerVolumeSpecName "kube-api-access-bqmqm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:22:05.019656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:05.019568    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqmqm\" (UniqueName: \"kubernetes.io/projected/99d2f743-ff01-4f3a-a4f0-d48467661ea3-kube-api-access-bqmqm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:05.019656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:05.019602    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/99d2f743-ff01-4f3a-a4f0-d48467661ea3-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:05.019656 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:05.019612    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/99d2f743-ff01-4f3a-a4f0-d48467661ea3-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:05.166585 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:05.166550    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"]
Nov 03 01:22:05.169855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:05.169829    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-564dd8d759-x4mlh"]
Nov 03 01:22:06.566561 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:06.566530    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d2f743-ff01-4f3a-a4f0-d48467661ea3" path="/var/lib/kubelet/pods/99d2f743-ff01-4f3a-a4f0-d48467661ea3/volumes"
Nov 03 01:22:11.231448 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:11.231408    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:22:11.231863 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:11.231522    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:22:13.617592 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:13.617561    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:22:19.895414 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:22:19.895380    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:22:21.231563 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:21.231525    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:22:25.504670 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.504637    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"]
Nov 03 01:22:25.505184 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.505166    6000 cpu_manager.go:414] "containerMap: removing container" podUID="99d2f743-ff01-4f3a-a4f0-d48467661ea3" containerName="main"
Nov 03 01:22:25.505234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.505188    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d2f743-ff01-4f3a-a4f0-d48467661ea3" containerName="main"
Nov 03 01:22:25.505351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.505339    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d2f743-ff01-4f3a-a4f0-d48467661ea3" containerName="main"
Nov 03 01:22:25.508671 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.508646    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.527423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.527394    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"]
Nov 03 01:22:25.537358 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.537331    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b9d7ac\"/\"openshift-service-ca.crt\""
Nov 03 01:22:25.537523 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.537418    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b9d7ac\"/\"default-dockercfg-bc7h9\""
Nov 03 01:22:25.537608 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.537532    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-b9d7ac\"/\"simulator-single-main-b9d7ac-kserve-self-signed-certs\""
Nov 03 01:22:25.537608 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.537589    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-b9d7ac\"/\"kube-root-ca.crt\""
Nov 03 01:22:25.636538 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.636503    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tls-certs\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.636538 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.636542    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tmp-dir\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.636786 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.636675    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-model-cache\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.637240 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.637211    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lhws\" (UniqueName: \"kubernetes.io/projected/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-kube-api-access-8lhws\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.637421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.637376    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-home\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.637421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.637418    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-dshm\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.738807 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.738763    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lhws\" (UniqueName: \"kubernetes.io/projected/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-kube-api-access-8lhws\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.739044 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.738823    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-home\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.739044 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.738873    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-dshm\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.739044 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.738918    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tls-certs\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.739044 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.738977    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tmp-dir\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.739316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.739106    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-model-cache\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.739646 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.739626    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tmp-dir\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.739805 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.739771    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-model-cache\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.740664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.740636    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-home\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.743212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.743187    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-dshm\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.756983 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.756906    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tls-certs\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.777917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.777888    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lhws\" (UniqueName: \"kubernetes.io/projected/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-kube-api-access-8lhws\") pod \"simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") " pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:25.821022 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:25.820996    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:26.116574 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:26.116541    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"]
Nov 03 01:22:26.117915 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:22:26.117883    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ea16c9f_c49f_4f56_8af8_15625e45b5f8.slice/crio-87c4c5048765419317bee826a9ff504d7725ad482fe6fc78155607507eec89f3 WatchSource:0}: Error finding container 87c4c5048765419317bee826a9ff504d7725ad482fe6fc78155607507eec89f3: Status 404 returned error can't find the container with id 87c4c5048765419317bee826a9ff504d7725ad482fe6fc78155607507eec89f3
Nov 03 01:22:26.920921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:26.920888    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8" event={"ID":"6ea16c9f-c49f-4f56-8af8-15625e45b5f8","Type":"ContainerStarted","Data":"22a35337d9b091bb735ab4b479c1177f3cdf8bc26b5f0b010fe4ee6de1466d09"}
Nov 03 01:22:26.921335 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:26.920926    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8" event={"ID":"6ea16c9f-c49f-4f56-8af8-15625e45b5f8","Type":"ContainerStarted","Data":"87c4c5048765419317bee826a9ff504d7725ad482fe6fc78155607507eec89f3"}
Nov 03 01:22:26.957309 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:26.957227    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8" podStartSLOduration=1.95720819 podStartE2EDuration="1.95720819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:22:26.957209433 +0000 UTC m=+4584.512561457" watchObservedRunningTime="2026-11-03 01:22:26.95720819 +0000 UTC m=+4584.512560217"
Nov 03 01:22:27.808946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:27.808912    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:22:27.821625 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:27.821605    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:31.231586 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:31.231546    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:22:34.662023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.661997    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:22:34.732759 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.732726    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-kube-api-access-8q64x\" (UniqueName: \"kubernetes.io/projected/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-kube-api-access-8q64x\") pod \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") "
Nov 03 01:22:34.732948 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.732806    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-tls-certs\" (UniqueName: \"kubernetes.io/secret/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-tls-certs\") pod \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\" (UID: \"71fd5526-2d22-4aaf-88ab-649b4e3c9c80\") "
Nov 03 01:22:34.735032 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.735002    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-tls-certs" pod "71fd5526-2d22-4aaf-88ab-649b4e3c9c80" (UID: "71fd5526-2d22-4aaf-88ab-649b4e3c9c80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:22:34.735182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.735097    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-kube-api-access-8q64x" pod "71fd5526-2d22-4aaf-88ab-649b4e3c9c80" (UID: "71fd5526-2d22-4aaf-88ab-649b4e3c9c80"). InnerVolumeSpecName "kube-api-access-8q64x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:22:34.833736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.833654    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:34.833736 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.833685    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8q64x\" (UniqueName: \"kubernetes.io/projected/71fd5526-2d22-4aaf-88ab-649b4e3c9c80-kube-api-access-8q64x\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:22:34.957926 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.957891    6000 generic.go:356] "Generic (PLEG): container finished" podID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerID="5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c" exitCode=0
Nov 03 01:22:34.958126 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.957947    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"
Nov 03 01:22:34.958126 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.957981    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" event={"ID":"71fd5526-2d22-4aaf-88ab-649b4e3c9c80","Type":"ContainerDied","Data":"5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c"}
Nov 03 01:22:34.958126 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.958037    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf" event={"ID":"71fd5526-2d22-4aaf-88ab-649b4e3c9c80","Type":"ContainerDied","Data":"2503bd2850fd38a453f1c72b89900cfda2e2dcd912c8fc3097e38112cb450b1d"}
Nov 03 01:22:34.958126 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.958062    6000 scope.go:122] "RemoveContainer" containerID="5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c"
Nov 03 01:22:34.967936 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.967916    6000 scope.go:122] "RemoveContainer" containerID="5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c"
Nov 03 01:22:34.968202 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:22:34.968181    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c\": container with ID starting with 5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c not found: ID does not exist" containerID="5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c"
Nov 03 01:22:34.968316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:34.968214    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c"} err="failed to get container status \"5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c\": rpc error: code = NotFound desc = could not find container \"5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c\": container with ID starting with 5a15b22d092a9f8b24de4f3c963f09bce18734bcb379822abbf217f56558b67c not found: ID does not exist"
Nov 03 01:22:35.005426 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:35.005393    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"]
Nov 03 01:22:35.008499 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:35.008472    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-prio-pool-a77df7ec-kserve-router-scheduler-86b8f66dc4-899bf"]
Nov 03 01:22:35.821965 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:35.821929    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:35.834840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:35.834816    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:35.974774 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:35.974743    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:22:36.565021 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:36.564989    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" path="/var/lib/kubelet/pods/71fd5526-2d22-4aaf-88ab-649b4e3c9c80/volumes"
Nov 03 01:22:43.698545 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:22:43.698446    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:22:49.896057 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:22:49.896018    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:23:07.757460 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:07.757419    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:23:12.028333 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:12.028301    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"]
Nov 03 01:23:17.739213 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:17.739175    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:23:19.895821 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:23:19.895772    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:23:27.049308 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.049247    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8" podUID="6ea16c9f-c49f-4f56-8af8-15625e45b5f8" containerName="main" containerID="cri-o://22a35337d9b091bb735ab4b479c1177f3cdf8bc26b5f0b010fe4ee6de1466d09" gracePeriod=45
Nov 03 01:23:27.218016 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.217981    6000 generic.go:356] "Generic (PLEG): container finished" podID="6ea16c9f-c49f-4f56-8af8-15625e45b5f8" containerID="22a35337d9b091bb735ab4b479c1177f3cdf8bc26b5f0b010fe4ee6de1466d09" exitCode=0
Nov 03 01:23:27.218207 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.218054    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8" event={"ID":"6ea16c9f-c49f-4f56-8af8-15625e45b5f8","Type":"ContainerDied","Data":"22a35337d9b091bb735ab4b479c1177f3cdf8bc26b5f0b010fe4ee6de1466d09"}
Nov 03 01:23:27.301433 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.301364    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:23:27.386480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386449    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tls-certs\") pod \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") "
Nov 03 01:23:27.386670 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386494    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-home\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-home\") pod \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") "
Nov 03 01:23:27.386670 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386542    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-kube-api-access-8lhws\" (UniqueName: \"kubernetes.io/projected/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-kube-api-access-8lhws\") pod \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") "
Nov 03 01:23:27.386670 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386603    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tmp-dir\") pod \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") "
Nov 03 01:23:27.386670 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386648    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-model-cache\") pod \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") "
Nov 03 01:23:27.386928 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386747    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-dshm\") pod \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\" (UID: \"6ea16c9f-c49f-4f56-8af8-15625e45b5f8\") "
Nov 03 01:23:27.386928 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386811    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-home" pod "6ea16c9f-c49f-4f56-8af8-15625e45b5f8" (UID: "6ea16c9f-c49f-4f56-8af8-15625e45b5f8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:27.387061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386970    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-model-cache" pod "6ea16c9f-c49f-4f56-8af8-15625e45b5f8" (UID: "6ea16c9f-c49f-4f56-8af8-15625e45b5f8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:27.387061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.386947    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tmp-dir" pod "6ea16c9f-c49f-4f56-8af8-15625e45b5f8" (UID: "6ea16c9f-c49f-4f56-8af8-15625e45b5f8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:27.387212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.387192    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:23:27.387314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.387215    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:23:27.387314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.387224    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:23:27.389083 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.389019    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-kube-api-access-8lhws" pod "6ea16c9f-c49f-4f56-8af8-15625e45b5f8" (UID: "6ea16c9f-c49f-4f56-8af8-15625e45b5f8"). InnerVolumeSpecName "kube-api-access-8lhws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:23:27.389205 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.389094    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-dshm" pod "6ea16c9f-c49f-4f56-8af8-15625e45b5f8" (UID: "6ea16c9f-c49f-4f56-8af8-15625e45b5f8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:23:27.389326 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.389310    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tls-certs" pod "6ea16c9f-c49f-4f56-8af8-15625e45b5f8" (UID: "6ea16c9f-c49f-4f56-8af8-15625e45b5f8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:23:27.488141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.488098    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:23:27.488141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.488133    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:23:27.488141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:27.488143    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lhws\" (UniqueName: \"kubernetes.io/projected/6ea16c9f-c49f-4f56-8af8-15625e45b5f8-kube-api-access-8lhws\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:23:28.225094 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:28.225060    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8" event={"ID":"6ea16c9f-c49f-4f56-8af8-15625e45b5f8","Type":"ContainerDied","Data":"87c4c5048765419317bee826a9ff504d7725ad482fe6fc78155607507eec89f3"}
Nov 03 01:23:28.225577 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:28.225111    6000 scope.go:122] "RemoveContainer" containerID="22a35337d9b091bb735ab4b479c1177f3cdf8bc26b5f0b010fe4ee6de1466d09"
Nov 03 01:23:28.225577 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:28.225069    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"
Nov 03 01:23:28.271860 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:28.271827    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"]
Nov 03 01:23:28.282789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:28.282758    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-b9d7ac/simulator-single-main-b9d7ac-kserve-9d9f99c94-zsjn8"]
Nov 03 01:23:28.565587 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:28.565505    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ea16c9f-c49f-4f56-8af8-15625e45b5f8" path="/var/lib/kubelet/pods/6ea16c9f-c49f-4f56-8af8-15625e45b5f8/volumes"
Nov 03 01:23:47.569743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.569704    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"]
Nov 03 01:23:47.570292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.570141    6000 cpu_manager.go:414] "containerMap: removing container" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main"
Nov 03 01:23:47.570292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.570154    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main"
Nov 03 01:23:47.570292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.570180    6000 cpu_manager.go:414] "containerMap: removing container" podUID="6ea16c9f-c49f-4f56-8af8-15625e45b5f8" containerName="main"
Nov 03 01:23:47.570292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.570185    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ea16c9f-c49f-4f56-8af8-15625e45b5f8" containerName="main"
Nov 03 01:23:47.570292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.570279    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="71fd5526-2d22-4aaf-88ab-649b4e3c9c80" containerName="main"
Nov 03 01:23:47.570292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.570291    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ea16c9f-c49f-4f56-8af8-15625e45b5f8" containerName="main"
Nov 03 01:23:47.573361 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.573341    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.599499 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.599460    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b9d7ac-serving-cert\""
Nov 03 01:23:47.608840 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.608792    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"]
Nov 03 01:23:47.694008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.693968    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76wgb\" (UniqueName: \"kubernetes.io/projected/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-kube-api-access-76wgb\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.694214 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.694083    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.694214 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.694119    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.694214 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.694165    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.794951 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.794902    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.795150 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.794996    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.795150 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.795052    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.795150 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.795125    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76wgb\" (UniqueName: \"kubernetes.io/projected/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-kube-api-access-76wgb\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.795934 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.795762    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.797838 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.797809    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.798846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.798817    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.813451 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.813421    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76wgb\" (UniqueName: \"kubernetes.io/projected/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-kube-api-access-76wgb\") pod \"maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.887012 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.886977    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:47.900271 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.900230    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"]
Nov 03 01:23:47.903971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.903950    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:47.917837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.917803    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-adf1237d-dockercfg-zwdsg\""
Nov 03 01:23:47.918891 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.918869    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-b9d7ac\""
Nov 03 01:23:47.923621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.923597    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"]
Nov 03 01:23:47.960679 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.960647    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"]
Nov 03 01:23:47.967541 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.966851    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:47.982903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:47.982823    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"]
Nov 03 01:23:48.099384 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.099346    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8b8sj\" (UniqueName: \"kubernetes.io/projected/284423cf-78b6-41c9-8198-ce008c530e6c-kube-api-access-8b8sj\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:48.099589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.099421    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef4cb91-743f-48f9-b627-9193560bb917-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:48.099589 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.099560    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mc5\" (UniqueName: \"kubernetes.io/projected/4ef4cb91-743f-48f9-b627-9193560bb917-kube-api-access-r5mc5\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:48.099695 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.099592    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284423cf-78b6-41c9-8198-ce008c530e6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:48.193562 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.193532    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"]
Nov 03 01:23:48.195507 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:23:48.195475    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ed3ba9e_5744_4a8c_aa96_ff093a47c222.slice/crio-dc40c4894c3d4e322734e8fb573180252522a20306f0b83f5b3b83727bb4fe0d WatchSource:0}: Error finding container dc40c4894c3d4e322734e8fb573180252522a20306f0b83f5b3b83727bb4fe0d: Status 404 returned error can't find the container with id dc40c4894c3d4e322734e8fb573180252522a20306f0b83f5b3b83727bb4fe0d
Nov 03 01:23:48.196991 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.196971    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:23:48.200214 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.200191    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mc5\" (UniqueName: \"kubernetes.io/projected/4ef4cb91-743f-48f9-b627-9193560bb917-kube-api-access-r5mc5\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:48.200317 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.200224    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284423cf-78b6-41c9-8198-ce008c530e6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:48.200369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.200316    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8b8sj\" (UniqueName: \"kubernetes.io/projected/284423cf-78b6-41c9-8198-ce008c530e6c-kube-api-access-8b8sj\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:48.200415 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.200387    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef4cb91-743f-48f9-b627-9193560bb917-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:48.201061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.201036    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284423cf-78b6-41c9-8198-ce008c530e6c-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:48.201184 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.201160    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef4cb91-743f-48f9-b627-9193560bb917-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:48.218619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.218592    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8b8sj\" (UniqueName: \"kubernetes.io/projected/284423cf-78b6-41c9-8198-ce008c530e6c-kube-api-access-8b8sj\") pod \"payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:48.218891 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.218874    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mc5\" (UniqueName: \"kubernetes.io/projected/4ef4cb91-743f-48f9-b627-9193560bb917-kube-api-access-r5mc5\") pod \"payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:48.228599 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.228575    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:48.286954 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.286921    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:48.320512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.320475    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr" event={"ID":"5ed3ba9e-5744-4a8c-aa96-ff093a47c222","Type":"ContainerStarted","Data":"dc40c4894c3d4e322734e8fb573180252522a20306f0b83f5b3b83727bb4fe0d"}
Nov 03 01:23:48.521594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.521548    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"]
Nov 03 01:23:48.523857 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:23:48.523828    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod284423cf_78b6_41c9_8198_ce008c530e6c.slice/crio-7edb8a569467558de195ffa283b1d9f83ce53286daa3c2d8e9991b8ff5f7a46e WatchSource:0}: Error finding container 7edb8a569467558de195ffa283b1d9f83ce53286daa3c2d8e9991b8ff5f7a46e: Status 404 returned error can't find the container with id 7edb8a569467558de195ffa283b1d9f83ce53286daa3c2d8e9991b8ff5f7a46e
Nov 03 01:23:48.584149 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:48.584120    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"]
Nov 03 01:23:48.587291 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:23:48.587240    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef4cb91_743f_48f9_b627_9193560bb917.slice/crio-6fb7290e0f0f7b15a97a164cd753d5a070f65f55e53b978d96bed0bb71b449dd WatchSource:0}: Error finding container 6fb7290e0f0f7b15a97a164cd753d5a070f65f55e53b978d96bed0bb71b449dd: Status 404 returned error can't find the container with id 6fb7290e0f0f7b15a97a164cd753d5a070f65f55e53b978d96bed0bb71b449dd
Nov 03 01:23:49.327615 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.327577    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr" event={"ID":"5ed3ba9e-5744-4a8c-aa96-ff093a47c222","Type":"ContainerStarted","Data":"bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8"}
Nov 03 01:23:49.327834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.327718    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:23:49.329061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.329035    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n" event={"ID":"284423cf-78b6-41c9-8198-ce008c530e6c","Type":"ContainerStarted","Data":"d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c"}
Nov 03 01:23:49.329190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.329084    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:23:49.329190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.329106    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n" event={"ID":"284423cf-78b6-41c9-8198-ce008c530e6c","Type":"ContainerStarted","Data":"7edb8a569467558de195ffa283b1d9f83ce53286daa3c2d8e9991b8ff5f7a46e"}
Nov 03 01:23:49.330392 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.330368    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd" event={"ID":"4ef4cb91-743f-48f9-b627-9193560bb917","Type":"ContainerStarted","Data":"bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee"}
Nov 03 01:23:49.330489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.330413    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd" event={"ID":"4ef4cb91-743f-48f9-b627-9193560bb917","Type":"ContainerStarted","Data":"6fb7290e0f0f7b15a97a164cd753d5a070f65f55e53b978d96bed0bb71b449dd"}
Nov 03 01:23:49.330489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.330440    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:23:49.364703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.364652    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr" podStartSLOduration=1.972669946 podStartE2EDuration="2.364636558s" totalImagesPullingTime="391.966612ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:49.363572916 +0000 UTC m=+4666.918924940" watchObservedRunningTime="2026-11-03 01:23:49.364636558 +0000 UTC m=+4666.919988585"
Nov 03 01:23:49.401497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.401436    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n" podStartSLOduration=2.401422145 podStartE2EDuration="2.401422145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:49.40056302 +0000 UTC m=+4666.955915040" watchObservedRunningTime="2026-11-03 01:23:49.401422145 +0000 UTC m=+4666.956774163"
Nov 03 01:23:49.437733 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:49.437683    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd" podStartSLOduration=2.437667981 podStartE2EDuration="2.437667981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:23:49.43544312 +0000 UTC m=+4666.990795144" watchObservedRunningTime="2026-11-03 01:23:49.437667981 +0000 UTC m=+4666.993019999"
Nov 03 01:23:49.894926 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:23:49.894882    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:23:51.334458 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:51.334420    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:23:55.341269 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:23:55.341234    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:24:00.338919 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:00.338882    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:24:00.339436 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:00.339308    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:24:06.507004 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:06.506968    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"]
Nov 03 01:24:06.540269 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:06.540230    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"]
Nov 03 01:24:07.868169 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:07.868132    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"]
Nov 03 01:24:07.868703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:07.868397    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr" podUID="5ed3ba9e-5744-4a8c-aa96-ff093a47c222" containerName="maas-api" containerID="cri-o://bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8" gracePeriod=30
Nov 03 01:24:08.123287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.123187    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:24:08.208071 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.208031    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-metrics-certs\") pod \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") "
Nov 03 01:24:08.208250 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.208114    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-kube-api-access-76wgb\" (UniqueName: \"kubernetes.io/projected/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-kube-api-access-76wgb\") pod \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") "
Nov 03 01:24:08.208250 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.208176    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-maas-api-tls\") pod \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") "
Nov 03 01:24:08.208425 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.208342    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-openshift-service-ca\") pod \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\" (UID: \"5ed3ba9e-5744-4a8c-aa96-ff093a47c222\") "
Nov 03 01:24:08.208769 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.208736    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-openshift-service-ca" pod "5ed3ba9e-5744-4a8c-aa96-ff093a47c222" (UID: "5ed3ba9e-5744-4a8c-aa96-ff093a47c222"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:08.210396 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.210373    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-metrics-certs" pod "5ed3ba9e-5744-4a8c-aa96-ff093a47c222" (UID: "5ed3ba9e-5744-4a8c-aa96-ff093a47c222"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:24:08.210524 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.210502    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-kube-api-access-76wgb" pod "5ed3ba9e-5744-4a8c-aa96-ff093a47c222" (UID: "5ed3ba9e-5744-4a8c-aa96-ff093a47c222"). InnerVolumeSpecName "kube-api-access-76wgb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:08.210570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.210507    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-maas-api-tls" pod "5ed3ba9e-5744-4a8c-aa96-ff093a47c222" (UID: "5ed3ba9e-5744-4a8c-aa96-ff093a47c222"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:24:08.310556 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.310516    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:08.310556 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.310549    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76wgb\" (UniqueName: \"kubernetes.io/projected/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-kube-api-access-76wgb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:08.310556 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.310559    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:08.310899 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.310569    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ed3ba9e-5744-4a8c-aa96-ff093a47c222-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:08.425797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.425704    6000 generic.go:356] "Generic (PLEG): container finished" podID="5ed3ba9e-5744-4a8c-aa96-ff093a47c222" containerID="bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8" exitCode=0
Nov 03 01:24:08.425797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.425752    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"
Nov 03 01:24:08.425797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.425765    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr" event={"ID":"5ed3ba9e-5744-4a8c-aa96-ff093a47c222","Type":"ContainerDied","Data":"bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8"}
Nov 03 01:24:08.426130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.425817    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr" event={"ID":"5ed3ba9e-5744-4a8c-aa96-ff093a47c222","Type":"ContainerDied","Data":"dc40c4894c3d4e322734e8fb573180252522a20306f0b83f5b3b83727bb4fe0d"}
Nov 03 01:24:08.426130 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.425844    6000 scope.go:122] "RemoveContainer" containerID="bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8"
Nov 03 01:24:08.435604 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.435582    6000 scope.go:122] "RemoveContainer" containerID="bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8"
Nov 03 01:24:08.435861 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:24:08.435840    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8\": container with ID starting with bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8 not found: ID does not exist" containerID="bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8"
Nov 03 01:24:08.435908 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.435871    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8"} err="failed to get container status \"bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8\": rpc error: code = NotFound desc = could not find container \"bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8\": container with ID starting with bcab0dd28c575d15a69e483347860bdb758c712db0d903f300ed5490bb3d1cc8 not found: ID does not exist"
Nov 03 01:24:08.472661 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.472625    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"]
Nov 03 01:24:08.476447 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.476421    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b9d7ac-5dddd9cf7b-5m8qr"]
Nov 03 01:24:08.569728 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:08.569694    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ed3ba9e-5744-4a8c-aa96-ff093a47c222" path="/var/lib/kubelet/pods/5ed3ba9e-5744-4a8c-aa96-ff093a47c222/volumes"
Nov 03 01:24:10.725903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:10.725865    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:24:11.507609 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.507551    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n" podUID="284423cf-78b6-41c9-8198-ce008c530e6c" containerName="payload-processing" containerID="cri-o://d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c" gracePeriod=25
Nov 03 01:24:11.541311 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.541244    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd" podUID="4ef4cb91-743f-48f9-b627-9193560bb917" containerName="payload-pre-processing" containerID="cri-o://bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee" gracePeriod=25
Nov 03 01:24:11.788376 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.788350    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:24:11.818814 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.818788    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:24:11.950480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.950444    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef4cb91-743f-48f9-b627-9193560bb917-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef4cb91-743f-48f9-b627-9193560bb917-plugins-config-volume\") pod \"4ef4cb91-743f-48f9-b627-9193560bb917\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") "
Nov 03 01:24:11.950480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.950486    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef4cb91-743f-48f9-b627-9193560bb917-kube-api-access-r5mc5\" (UniqueName: \"kubernetes.io/projected/4ef4cb91-743f-48f9-b627-9193560bb917-kube-api-access-r5mc5\") pod \"4ef4cb91-743f-48f9-b627-9193560bb917\" (UID: \"4ef4cb91-743f-48f9-b627-9193560bb917\") "
Nov 03 01:24:11.950757 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.950564    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/284423cf-78b6-41c9-8198-ce008c530e6c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284423cf-78b6-41c9-8198-ce008c530e6c-plugins-config-volume\") pod \"284423cf-78b6-41c9-8198-ce008c530e6c\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") "
Nov 03 01:24:11.950757 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.950589    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/284423cf-78b6-41c9-8198-ce008c530e6c-kube-api-access-8b8sj\" (UniqueName: \"kubernetes.io/projected/284423cf-78b6-41c9-8198-ce008c530e6c-kube-api-access-8b8sj\") pod \"284423cf-78b6-41c9-8198-ce008c530e6c\" (UID: \"284423cf-78b6-41c9-8198-ce008c530e6c\") "
Nov 03 01:24:11.950939 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.950913    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/284423cf-78b6-41c9-8198-ce008c530e6c-plugins-config-volume" pod "284423cf-78b6-41c9-8198-ce008c530e6c" (UID: "284423cf-78b6-41c9-8198-ce008c530e6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:11.950996 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.950917    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef4cb91-743f-48f9-b627-9193560bb917-plugins-config-volume" pod "4ef4cb91-743f-48f9-b627-9193560bb917" (UID: "4ef4cb91-743f-48f9-b627-9193560bb917"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:24:11.952802 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.952780    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef4cb91-743f-48f9-b627-9193560bb917-kube-api-access-r5mc5" pod "4ef4cb91-743f-48f9-b627-9193560bb917" (UID: "4ef4cb91-743f-48f9-b627-9193560bb917"). InnerVolumeSpecName "kube-api-access-r5mc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:11.952862 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:11.952795    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/284423cf-78b6-41c9-8198-ce008c530e6c-kube-api-access-8b8sj" pod "284423cf-78b6-41c9-8198-ce008c530e6c" (UID: "284423cf-78b6-41c9-8198-ce008c530e6c"). InnerVolumeSpecName "kube-api-access-8b8sj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:24:12.051712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.051599    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef4cb91-743f-48f9-b627-9193560bb917-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:12.051712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.051631    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mc5\" (UniqueName: \"kubernetes.io/projected/4ef4cb91-743f-48f9-b627-9193560bb917-kube-api-access-r5mc5\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:12.051712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.051646    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/284423cf-78b6-41c9-8198-ce008c530e6c-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:12.051712 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.051660    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8b8sj\" (UniqueName: \"kubernetes.io/projected/284423cf-78b6-41c9-8198-ce008c530e6c-kube-api-access-8b8sj\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:24:12.446613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.446485    6000 generic.go:356] "Generic (PLEG): container finished" podID="284423cf-78b6-41c9-8198-ce008c530e6c" containerID="d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c" exitCode=0
Nov 03 01:24:12.446613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.446528    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n" event={"ID":"284423cf-78b6-41c9-8198-ce008c530e6c","Type":"ContainerDied","Data":"d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c"}
Nov 03 01:24:12.446915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.446554    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"
Nov 03 01:24:12.446915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.446646    6000 scope.go:122] "RemoveContainer" containerID="d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c"
Nov 03 01:24:12.446915 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.446628    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n" event={"ID":"284423cf-78b6-41c9-8198-ce008c530e6c","Type":"ContainerDied","Data":"7edb8a569467558de195ffa283b1d9f83ce53286daa3c2d8e9991b8ff5f7a46e"}
Nov 03 01:24:12.447987 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.447960    6000 generic.go:356] "Generic (PLEG): container finished" podID="4ef4cb91-743f-48f9-b627-9193560bb917" containerID="bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee" exitCode=0
Nov 03 01:24:12.448086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.448007    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"
Nov 03 01:24:12.448086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.448006    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd" event={"ID":"4ef4cb91-743f-48f9-b627-9193560bb917","Type":"ContainerDied","Data":"bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee"}
Nov 03 01:24:12.448217 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.448108    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd" event={"ID":"4ef4cb91-743f-48f9-b627-9193560bb917","Type":"ContainerDied","Data":"6fb7290e0f0f7b15a97a164cd753d5a070f65f55e53b978d96bed0bb71b449dd"}
Nov 03 01:24:12.455880 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.455864    6000 scope.go:122] "RemoveContainer" containerID="d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c"
Nov 03 01:24:12.456170 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:24:12.456151    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c\": container with ID starting with d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c not found: ID does not exist" containerID="d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c"
Nov 03 01:24:12.456340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.456179    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c"} err="failed to get container status \"d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c\": rpc error: code = NotFound desc = could not find container \"d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c\": container with ID starting with d57bbb5e296fa80700c426ba8a1951c9b500705902d19b418b277d04418f9b7c not found: ID does not exist"
Nov 03 01:24:12.456340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.456195    6000 scope.go:122] "RemoveContainer" containerID="bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee"
Nov 03 01:24:12.465039 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.465022    6000 scope.go:122] "RemoveContainer" containerID="bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee"
Nov 03 01:24:12.465343 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:24:12.465325    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee\": container with ID starting with bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee not found: ID does not exist" containerID="bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee"
Nov 03 01:24:12.465406 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.465351    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee"} err="failed to get container status \"bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee\": rpc error: code = NotFound desc = could not find container \"bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee\": container with ID starting with bf7dceff81ff326fddb66b2a46b7b511ad4300870dcf65f18f62ce915d1ba2ee not found: ID does not exist"
Nov 03 01:24:12.498181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.498153    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"]
Nov 03 01:24:12.504229 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.504207    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-b9d7ac-f86f8g8ghd"]
Nov 03 01:24:12.534976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.534949    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"]
Nov 03 01:24:12.538939 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.538915    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-b9d7ac-68d9d486dztz4n"]
Nov 03 01:24:12.565856 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.565831    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="284423cf-78b6-41c9-8198-ce008c530e6c" path="/var/lib/kubelet/pods/284423cf-78b6-41c9-8198-ce008c530e6c/volumes"
Nov 03 01:24:12.566233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:12.566217    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef4cb91-743f-48f9-b627-9193560bb917" path="/var/lib/kubelet/pods/4ef4cb91-743f-48f9-b627-9193560bb917/volumes"
Nov 03 01:24:13.965688 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.965650    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"]
Nov 03 01:24:13.966182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966164    6000 cpu_manager.go:414] "containerMap: removing container" podUID="5ed3ba9e-5744-4a8c-aa96-ff093a47c222" containerName="maas-api"
Nov 03 01:24:13.966233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966184    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ed3ba9e-5744-4a8c-aa96-ff093a47c222" containerName="maas-api"
Nov 03 01:24:13.966233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966201    6000 cpu_manager.go:414] "containerMap: removing container" podUID="4ef4cb91-743f-48f9-b627-9193560bb917" containerName="payload-pre-processing"
Nov 03 01:24:13.966233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966206    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef4cb91-743f-48f9-b627-9193560bb917" containerName="payload-pre-processing"
Nov 03 01:24:13.966233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966224    6000 cpu_manager.go:414] "containerMap: removing container" podUID="284423cf-78b6-41c9-8198-ce008c530e6c" containerName="payload-processing"
Nov 03 01:24:13.966233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966229    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="284423cf-78b6-41c9-8198-ce008c530e6c" containerName="payload-processing"
Nov 03 01:24:13.966441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966305    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ed3ba9e-5744-4a8c-aa96-ff093a47c222" containerName="maas-api"
Nov 03 01:24:13.966441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966313    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef4cb91-743f-48f9-b627-9193560bb917" containerName="payload-pre-processing"
Nov 03 01:24:13.966441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.966322    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="284423cf-78b6-41c9-8198-ce008c530e6c" containerName="payload-processing"
Nov 03 01:24:13.969809 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.969776    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:13.995043 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.995013    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-e3903940-kserve-self-signed-certs\""
Nov 03 01:24:13.996694 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:13.996669    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"]
Nov 03 01:24:14.071304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.071245    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496ed215-7590-4071-ab86-0bf694b0f3e8-tls-certs\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.071459 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.071316    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-tmp-dir\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.071459 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.071426    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-model-cache\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.071556 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.071458    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-home\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.071556 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.071485    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vznws\" (UniqueName: \"kubernetes.io/projected/496ed215-7590-4071-ab86-0bf694b0f3e8-kube-api-access-vznws\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.071630 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.071610    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-dshm\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.172989    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496ed215-7590-4071-ab86-0bf694b0f3e8-tls-certs\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173240 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173056    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-tmp-dir\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173240 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173135    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-model-cache\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173240 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173164    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-home\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173240 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173213    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vznws\" (UniqueName: \"kubernetes.io/projected/496ed215-7590-4071-ab86-0bf694b0f3e8-kube-api-access-vznws\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173521 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173312    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-dshm\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173584 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173554    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-tmp-dir\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173638 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173590    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-model-cache\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.173971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.173909    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-home\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.175764 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.175741    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-dshm\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.176000 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.175978    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496ed215-7590-4071-ab86-0bf694b0f3e8-tls-certs\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.193212 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.193181    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vznws\" (UniqueName: \"kubernetes.io/projected/496ed215-7590-4071-ab86-0bf694b0f3e8-kube-api-access-vznws\") pod \"e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") " pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.284303 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.284199    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:14.578495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:14.578468    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"]
Nov 03 01:24:14.579856 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:24:14.579823    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod496ed215_7590_4071_ab86_0bf694b0f3e8.slice/crio-393e48285ab042298829e8d83db30b5d2c4b524ebdf6a3be1a35ede28658cfb8 WatchSource:0}: Error finding container 393e48285ab042298829e8d83db30b5d2c4b524ebdf6a3be1a35ede28658cfb8: Status 404 returned error can't find the container with id 393e48285ab042298829e8d83db30b5d2c4b524ebdf6a3be1a35ede28658cfb8
Nov 03 01:24:15.466654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:15.466617    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv" event={"ID":"496ed215-7590-4071-ab86-0bf694b0f3e8","Type":"ContainerStarted","Data":"734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f"}
Nov 03 01:24:15.466654 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:15.466660    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv" event={"ID":"496ed215-7590-4071-ab86-0bf694b0f3e8","Type":"ContainerStarted","Data":"393e48285ab042298829e8d83db30b5d2c4b524ebdf6a3be1a35ede28658cfb8"}
Nov 03 01:24:15.503011 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:15.502957    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv" podStartSLOduration=2.502943214 podStartE2EDuration="2.502943214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:15.501606518 +0000 UTC m=+4693.056958542" watchObservedRunningTime="2026-11-03 01:24:15.502943214 +0000 UTC m=+4693.058295237"
Nov 03 01:24:16.117183 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:16.117142    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:24:16.284451 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:16.284410    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:19.896915 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:24:19.896874    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:24:24.284610 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:24.284571    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:24.297809 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:24.297780    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:24.547023 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:24.546930    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:24:29.552383 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.552345    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"]
Nov 03 01:24:29.556439 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.556417    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.574542 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.574512    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"]
Nov 03 01:24:29.584441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.584405    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-e3903940-kserve-self-signed-certs\""
Nov 03 01:24:29.685791 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.685756    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-dshm\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.685963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.685809    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-tmp-dir\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.685963 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.685951    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfac29a-cb37-4d28-9333-2974af673023-tls-certs\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.686047 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.685984    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkwjn\" (UniqueName: \"kubernetes.io/projected/2dfac29a-cb37-4d28-9333-2974af673023-kube-api-access-lkwjn\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.686100 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.686088    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-model-cache\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.686141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.686130    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-home\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.786826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.786784    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-model-cache\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.786839    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-home\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.786864    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-dshm\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.786884    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-tmp-dir\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.786937    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfac29a-cb37-4d28-9333-2974af673023-tls-certs\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787033 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.786967    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkwjn\" (UniqueName: \"kubernetes.io/projected/2dfac29a-cb37-4d28-9333-2974af673023-kube-api-access-lkwjn\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.787207    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-model-cache\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787369 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.787322    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-home\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.787457 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.787433    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-tmp-dir\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.789387 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.789365    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-dshm\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.789808 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.789786    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfac29a-cb37-4d28-9333-2974af673023-tls-certs\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.805887 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.805826    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkwjn\" (UniqueName: \"kubernetes.io/projected/2dfac29a-cb37-4d28-9333-2974af673023-kube-api-access-lkwjn\") pod \"e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") " pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:29.874076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:29.874041    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:30.169091 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:30.169062    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"]
Nov 03 01:24:30.170775 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:24:30.170746    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2dfac29a_cb37_4d28_9333_2974af673023.slice/crio-0ff2166b616407e6ead4f1321d7b52df9ff529827a3a107fbe864b856e23e3c7 WatchSource:0}: Error finding container 0ff2166b616407e6ead4f1321d7b52df9ff529827a3a107fbe864b856e23e3c7: Status 404 returned error can't find the container with id 0ff2166b616407e6ead4f1321d7b52df9ff529827a3a107fbe864b856e23e3c7
Nov 03 01:24:30.567946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:30.567859    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz" event={"ID":"2dfac29a-cb37-4d28-9333-2974af673023","Type":"ContainerStarted","Data":"2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65"}
Nov 03 01:24:30.567946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:30.567899    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz" event={"ID":"2dfac29a-cb37-4d28-9333-2974af673023","Type":"ContainerStarted","Data":"0ff2166b616407e6ead4f1321d7b52df9ff529827a3a107fbe864b856e23e3c7"}
Nov 03 01:24:30.608298 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:30.608236    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz" podStartSLOduration=1.6082234789999998 podStartE2EDuration="1.608223479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:24:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:24:30.606250336 +0000 UTC m=+4708.161602359" watchObservedRunningTime="2026-11-03 01:24:30.608223479 +0000 UTC m=+4708.163575503"
Nov 03 01:24:30.874333 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:30.874298    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:32.045664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:32.045624    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:24:39.874790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:39.874751    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:39.886783 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:39.886754    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:40.676876 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:40.676846    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:24:48.340477 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:24:48.340441    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:24:49.910587 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:24:49.910522    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:25:00.411807 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:00.411773    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"]
Nov 03 01:25:01.827435 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:01.827389    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"]
Nov 03 01:25:02.420177 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:02.420140    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:25:08.612291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:08.611425    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:25:08.612291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:08.611508    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:25:08.706281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:08.704341    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-dhv97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:25:08.706281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:08.704431    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-dhv97" podUID="ad946480-1aaa-4a6c-826c-a0db0c4a9ac9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.55:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:25:08.983162 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:08.983062    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:25:15.433636 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.433576    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv" podUID="496ed215-7590-4071-ab86-0bf694b0f3e8" containerName="main" containerID="cri-o://734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f" gracePeriod=45
Nov 03 01:25:15.536228 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.536186    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv" podUID="496ed215-7590-4071-ab86-0bf694b0f3e8" containerName="main" probeResult="failure" output="Get \"https://10.134.1.64:8000/ready\": dial tcp 10.134.1.64:8000: connect: connection refused"
Nov 03 01:25:15.699219 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.699157    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:25:15.882591 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882546    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-tmp-dir\") pod \"496ed215-7590-4071-ab86-0bf694b0f3e8\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") "
Nov 03 01:25:15.882804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882633    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-dshm\") pod \"496ed215-7590-4071-ab86-0bf694b0f3e8\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") "
Nov 03 01:25:15.882804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882725    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-model-cache\") pod \"496ed215-7590-4071-ab86-0bf694b0f3e8\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") "
Nov 03 01:25:15.882804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882777    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-home\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-home\") pod \"496ed215-7590-4071-ab86-0bf694b0f3e8\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") "
Nov 03 01:25:15.882804 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882805    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496ed215-7590-4071-ab86-0bf694b0f3e8-kube-api-access-vznws\" (UniqueName: \"kubernetes.io/projected/496ed215-7590-4071-ab86-0bf694b0f3e8-kube-api-access-vznws\") pod \"496ed215-7590-4071-ab86-0bf694b0f3e8\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") "
Nov 03 01:25:15.883060 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882827    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/496ed215-7590-4071-ab86-0bf694b0f3e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/496ed215-7590-4071-ab86-0bf694b0f3e8-tls-certs\") pod \"496ed215-7590-4071-ab86-0bf694b0f3e8\" (UID: \"496ed215-7590-4071-ab86-0bf694b0f3e8\") "
Nov 03 01:25:15.883060 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882848    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-tmp-dir" pod "496ed215-7590-4071-ab86-0bf694b0f3e8" (UID: "496ed215-7590-4071-ab86-0bf694b0f3e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:15.883060 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.882966    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-model-cache" pod "496ed215-7590-4071-ab86-0bf694b0f3e8" (UID: "496ed215-7590-4071-ab86-0bf694b0f3e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:15.883060 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.883047    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-home" pod "496ed215-7590-4071-ab86-0bf694b0f3e8" (UID: "496ed215-7590-4071-ab86-0bf694b0f3e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:15.883247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.883190    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:15.883247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.883204    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:15.883247 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.883212    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:15.885095 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.885071    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/496ed215-7590-4071-ab86-0bf694b0f3e8-tls-certs" pod "496ed215-7590-4071-ab86-0bf694b0f3e8" (UID: "496ed215-7590-4071-ab86-0bf694b0f3e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:25:15.885095 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.885089    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-dshm" pod "496ed215-7590-4071-ab86-0bf694b0f3e8" (UID: "496ed215-7590-4071-ab86-0bf694b0f3e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:15.885309 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.885097    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496ed215-7590-4071-ab86-0bf694b0f3e8-kube-api-access-vznws" pod "496ed215-7590-4071-ab86-0bf694b0f3e8" (UID: "496ed215-7590-4071-ab86-0bf694b0f3e8"). InnerVolumeSpecName "kube-api-access-vznws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:25:15.927428 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.927398    6000 generic.go:356] "Generic (PLEG): container finished" podID="496ed215-7590-4071-ab86-0bf694b0f3e8" containerID="734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f" exitCode=0
Nov 03 01:25:15.927572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.927439    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv" event={"ID":"496ed215-7590-4071-ab86-0bf694b0f3e8","Type":"ContainerDied","Data":"734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f"}
Nov 03 01:25:15.927572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.927454    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"
Nov 03 01:25:15.927572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.927466    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv" event={"ID":"496ed215-7590-4071-ab86-0bf694b0f3e8","Type":"ContainerDied","Data":"393e48285ab042298829e8d83db30b5d2c4b524ebdf6a3be1a35ede28658cfb8"}
Nov 03 01:25:15.927572 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.927482    6000 scope.go:122] "RemoveContainer" containerID="734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f"
Nov 03 01:25:15.937009 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.936993    6000 scope.go:122] "RemoveContainer" containerID="734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f"
Nov 03 01:25:15.937325 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:25:15.937289    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f\": container with ID starting with 734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f not found: ID does not exist" containerID="734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f"
Nov 03 01:25:15.937325 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.937316    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f"} err="failed to get container status \"734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f\": rpc error: code = NotFound desc = could not find container \"734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f\": container with ID starting with 734bbb0e2b6f139244542c43ae78ff9270711b1026ce960015114d0f0e64c36f not found: ID does not exist"
Nov 03 01:25:15.975128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.975064    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"]
Nov 03 01:25:15.976677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.976658    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-e3903940-kserve-57bf6cdf89-vz8pv"]
Nov 03 01:25:15.983855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.983835    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vznws\" (UniqueName: \"kubernetes.io/projected/496ed215-7590-4071-ab86-0bf694b0f3e8-kube-api-access-vznws\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:15.983928 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.983860    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/496ed215-7590-4071-ab86-0bf694b0f3e8-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:15.983928 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:15.983873    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/496ed215-7590-4071-ab86-0bf694b0f3e8-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:16.565645 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:16.565610    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496ed215-7590-4071-ab86-0bf694b0f3e8" path="/var/lib/kubelet/pods/496ed215-7590-4071-ab86-0bf694b0f3e8/volumes"
Nov 03 01:25:16.856514 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:16.856459    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz" podUID="2dfac29a-cb37-4d28-9333-2974af673023" containerName="main" containerID="cri-o://2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65" gracePeriod=45
Nov 03 01:25:17.133293 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.133245    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:25:17.196224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196188    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dfac29a-cb37-4d28-9333-2974af673023-tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfac29a-cb37-4d28-9333-2974af673023-tls-certs\") pod \"2dfac29a-cb37-4d28-9333-2974af673023\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") "
Nov 03 01:25:17.196411 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196243    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-tmp-dir\") pod \"2dfac29a-cb37-4d28-9333-2974af673023\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") "
Nov 03 01:25:17.196411 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196316    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-home\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-home\") pod \"2dfac29a-cb37-4d28-9333-2974af673023\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") "
Nov 03 01:25:17.196411 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196392    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dfac29a-cb37-4d28-9333-2974af673023-kube-api-access-lkwjn\" (UniqueName: \"kubernetes.io/projected/2dfac29a-cb37-4d28-9333-2974af673023-kube-api-access-lkwjn\") pod \"2dfac29a-cb37-4d28-9333-2974af673023\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") "
Nov 03 01:25:17.196603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196459    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-dshm\") pod \"2dfac29a-cb37-4d28-9333-2974af673023\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") "
Nov 03 01:25:17.196603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196522    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-tmp-dir" pod "2dfac29a-cb37-4d28-9333-2974af673023" (UID: "2dfac29a-cb37-4d28-9333-2974af673023"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:17.196603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196539    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-model-cache\") pod \"2dfac29a-cb37-4d28-9333-2974af673023\" (UID: \"2dfac29a-cb37-4d28-9333-2974af673023\") "
Nov 03 01:25:17.196776 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196746    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-model-cache" pod "2dfac29a-cb37-4d28-9333-2974af673023" (UID: "2dfac29a-cb37-4d28-9333-2974af673023"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:17.196946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196905    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-home" pod "2dfac29a-cb37-4d28-9333-2974af673023" (UID: "2dfac29a-cb37-4d28-9333-2974af673023"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:17.197041 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.196991    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:17.197041 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.197010    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:17.198597 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.198571    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dfac29a-cb37-4d28-9333-2974af673023-tls-certs" pod "2dfac29a-cb37-4d28-9333-2974af673023" (UID: "2dfac29a-cb37-4d28-9333-2974af673023"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:25:17.198811 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.198794    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dfac29a-cb37-4d28-9333-2974af673023-kube-api-access-lkwjn" pod "2dfac29a-cb37-4d28-9333-2974af673023" (UID: "2dfac29a-cb37-4d28-9333-2974af673023"). InnerVolumeSpecName "kube-api-access-lkwjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:25:17.199187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.199161    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-dshm" pod "2dfac29a-cb37-4d28-9333-2974af673023" (UID: "2dfac29a-cb37-4d28-9333-2974af673023"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:25:17.297725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.297687    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkwjn\" (UniqueName: \"kubernetes.io/projected/2dfac29a-cb37-4d28-9333-2974af673023-kube-api-access-lkwjn\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:17.297725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.297718    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:17.297725 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.297732    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2dfac29a-cb37-4d28-9333-2974af673023-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:17.297971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.297744    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2dfac29a-cb37-4d28-9333-2974af673023-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:17.541005 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.540907    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:25:17.942485 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.942440    6000 generic.go:356] "Generic (PLEG): container finished" podID="2dfac29a-cb37-4d28-9333-2974af673023" containerID="2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65" exitCode=0
Nov 03 01:25:17.943076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.942501    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"
Nov 03 01:25:17.943076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.942540    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz" event={"ID":"2dfac29a-cb37-4d28-9333-2974af673023","Type":"ContainerDied","Data":"2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65"}
Nov 03 01:25:17.943076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.942593    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz" event={"ID":"2dfac29a-cb37-4d28-9333-2974af673023","Type":"ContainerDied","Data":"0ff2166b616407e6ead4f1321d7b52df9ff529827a3a107fbe864b856e23e3c7"}
Nov 03 01:25:17.943076 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.942616    6000 scope.go:122] "RemoveContainer" containerID="2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65"
Nov 03 01:25:17.955541 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.955470    6000 scope.go:122] "RemoveContainer" containerID="2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65"
Nov 03 01:25:17.955818 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:25:17.955784    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65\": container with ID starting with 2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65 not found: ID does not exist" containerID="2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65"
Nov 03 01:25:17.955896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.955830    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65"} err="failed to get container status \"2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65\": rpc error: code = NotFound desc = could not find container \"2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65\": container with ID starting with 2e8fc88d5c542d58db4f8c61af144bfeffc2c1949e0df05d606f2428d69b8a65 not found: ID does not exist"
Nov 03 01:25:17.992233 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:17.992200    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"]
Nov 03 01:25:18.009749 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:18.009675    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-e3903940-kserve-6ccffbd59c-6jfkz"]
Nov 03 01:25:18.579986 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:18.578008    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dfac29a-cb37-4d28-9333-2974af673023" path="/var/lib/kubelet/pods/2dfac29a-cb37-4d28-9333-2974af673023/volumes"
Nov 03 01:25:19.969959 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:25:19.969843    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:25:22.216783 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:22.216744    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:25:37.534441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:37.534403    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:25:41.944354 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.944319    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"]
Nov 03 01:25:41.944997 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.944975    6000 cpu_manager.go:414] "containerMap: removing container" podUID="496ed215-7590-4071-ab86-0bf694b0f3e8" containerName="main"
Nov 03 01:25:41.945102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.944999    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496ed215-7590-4071-ab86-0bf694b0f3e8" containerName="main"
Nov 03 01:25:41.945102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.945023    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2dfac29a-cb37-4d28-9333-2974af673023" containerName="main"
Nov 03 01:25:41.945102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.945030    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dfac29a-cb37-4d28-9333-2974af673023" containerName="main"
Nov 03 01:25:41.945304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.945180    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dfac29a-cb37-4d28-9333-2974af673023" containerName="main"
Nov 03 01:25:41.945304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.945207    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="496ed215-7590-4071-ab86-0bf694b0f3e8" containerName="main"
Nov 03 01:25:41.949631 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.949607    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:41.965642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.965616    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"]
Nov 03 01:25:41.975396 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:41.975375    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-csh4z\""
Nov 03 01:25:42.083707 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.083619    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562d41b3-f186-4131-a4b1-8ac819e24b1d-plugins-config-volume\") pod \"payload-pre-processing-b5668cfcb-2xmwb\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") " pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:42.083864 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.083811    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/562d41b3-f186-4131-a4b1-8ac819e24b1d-kube-api-access-jqvds\") pod \"payload-pre-processing-b5668cfcb-2xmwb\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") " pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:42.184951 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.184900    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/562d41b3-f186-4131-a4b1-8ac819e24b1d-kube-api-access-jqvds\") pod \"payload-pre-processing-b5668cfcb-2xmwb\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") " pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:42.185167 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.184994    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562d41b3-f186-4131-a4b1-8ac819e24b1d-plugins-config-volume\") pod \"payload-pre-processing-b5668cfcb-2xmwb\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") " pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:42.185764 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.185742    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562d41b3-f186-4131-a4b1-8ac819e24b1d-plugins-config-volume\") pod \"payload-pre-processing-b5668cfcb-2xmwb\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") " pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:42.203335 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.203244    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/562d41b3-f186-4131-a4b1-8ac819e24b1d-kube-api-access-jqvds\") pod \"payload-pre-processing-b5668cfcb-2xmwb\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") " pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:42.263466 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.263437    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:42.554008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:42.553978    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"]
Nov 03 01:25:42.554857 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:25:42.554829    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod562d41b3_f186_4131_a4b1_8ac819e24b1d.slice/crio-4f986385a2d7229debbd109742af515cb32b5ef2d35f81671272cd60e34f0d8f WatchSource:0}: Error finding container 4f986385a2d7229debbd109742af515cb32b5ef2d35f81671272cd60e34f0d8f: Status 404 returned error can't find the container with id 4f986385a2d7229debbd109742af515cb32b5ef2d35f81671272cd60e34f0d8f
Nov 03 01:25:43.099586 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:43.099551    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb" event={"ID":"562d41b3-f186-4131-a4b1-8ac819e24b1d","Type":"ContainerStarted","Data":"f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf"}
Nov 03 01:25:43.100151 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:43.099604    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb" event={"ID":"562d41b3-f186-4131-a4b1-8ac819e24b1d","Type":"ContainerStarted","Data":"4f986385a2d7229debbd109742af515cb32b5ef2d35f81671272cd60e34f0d8f"}
Nov 03 01:25:43.100151 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:43.099635    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:43.135453 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:43.135399    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb" podStartSLOduration=2.13538306 podStartE2EDuration="2.13538306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:43.135245689 +0000 UTC m=+4780.690597736" watchObservedRunningTime="2026-11-03 01:25:43.13538306 +0000 UTC m=+4780.690735063"
Nov 03 01:25:49.895282 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:25:49.895215    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:25:54.107677 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:54.107645    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:25:54.172998 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:54.172960    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"]
Nov 03 01:25:55.851620 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:55.851587    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-b7f977d8-nps9f"]
Nov 03 01:25:55.855461 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:55.855440    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:55.873422 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:55.873395    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b7f977d8-nps9f"]
Nov 03 01:25:55.916230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:55.916203    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed665f9-5ffc-438c-a98e-c52393835dc2-plugins-config-volume\") pod \"payload-processing-b7f977d8-nps9f\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") " pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:55.916230 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:55.916236    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcb6g\" (UniqueName: \"kubernetes.io/projected/aed665f9-5ffc-438c-a98e-c52393835dc2-kube-api-access-jcb6g\") pod \"payload-processing-b7f977d8-nps9f\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") " pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:56.016892 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:56.016860    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed665f9-5ffc-438c-a98e-c52393835dc2-plugins-config-volume\") pod \"payload-processing-b7f977d8-nps9f\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") " pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:56.017060 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:56.016900    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcb6g\" (UniqueName: \"kubernetes.io/projected/aed665f9-5ffc-438c-a98e-c52393835dc2-kube-api-access-jcb6g\") pod \"payload-processing-b7f977d8-nps9f\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") " pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:56.017606 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:56.017586    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed665f9-5ffc-438c-a98e-c52393835dc2-plugins-config-volume\") pod \"payload-processing-b7f977d8-nps9f\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") " pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:56.036032 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:56.036002    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcb6g\" (UniqueName: \"kubernetes.io/projected/aed665f9-5ffc-438c-a98e-c52393835dc2-kube-api-access-jcb6g\") pod \"payload-processing-b7f977d8-nps9f\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") " pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:56.168001 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:56.167922    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:56.461864 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:56.461834    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-b7f977d8-nps9f"]
Nov 03 01:25:56.463181 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:25:56.463146    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed665f9_5ffc_438c_a98e_c52393835dc2.slice/crio-183a53cc659b85b87c85b8528dd17a1c9c6f91428d719f3224be44c7950ecc22 WatchSource:0}: Error finding container 183a53cc659b85b87c85b8528dd17a1c9c6f91428d719f3224be44c7950ecc22: Status 404 returned error can't find the container with id 183a53cc659b85b87c85b8528dd17a1c9c6f91428d719f3224be44c7950ecc22
Nov 03 01:25:57.165849 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:57.165811    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b7f977d8-nps9f" event={"ID":"aed665f9-5ffc-438c-a98e-c52393835dc2","Type":"ContainerStarted","Data":"367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7"}
Nov 03 01:25:57.166323 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:57.165863    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b7f977d8-nps9f" event={"ID":"aed665f9-5ffc-438c-a98e-c52393835dc2","Type":"ContainerStarted","Data":"183a53cc659b85b87c85b8528dd17a1c9c6f91428d719f3224be44c7950ecc22"}
Nov 03 01:25:57.166323 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:57.165951    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:25:57.201774 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:57.201723    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-b7f977d8-nps9f" podStartSLOduration=2.201699582 podStartE2EDuration="2.201699582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:25:57.201492772 +0000 UTC m=+4794.756844797" watchObservedRunningTime="2026-11-03 01:25:57.201699582 +0000 UTC m=+4794.757051605"
Nov 03 01:25:59.173929 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.173886    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6" podUID="2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" containerName="payload-pre-processing" containerID="cri-o://27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2" gracePeriod=25
Nov 03 01:25:59.426665 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.426605    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 01:25:59.448988 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.448960    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-plugins-config-volume\") pod \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") "
Nov 03 01:25:59.449183 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.448993    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-kube-api-access-t65qw\" (UniqueName: \"kubernetes.io/projected/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-kube-api-access-t65qw\") pod \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\" (UID: \"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe\") "
Nov 03 01:25:59.449357 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.449334    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-plugins-config-volume" pod "2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" (UID: "2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:25:59.451249 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.451227    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-kube-api-access-t65qw" pod "2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" (UID: "2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe"). InnerVolumeSpecName "kube-api-access-t65qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:25:59.550670 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.550633    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:25:59.550670 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:25:59.550665    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t65qw\" (UniqueName: \"kubernetes.io/projected/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe-kube-api-access-t65qw\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:26:00.181893 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.181854    6000 generic.go:356] "Generic (PLEG): container finished" podID="2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" containerID="27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2" exitCode=0
Nov 03 01:26:00.182394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.181923    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"
Nov 03 01:26:00.182394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.181938    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6" event={"ID":"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe","Type":"ContainerDied","Data":"27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2"}
Nov 03 01:26:00.182394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.182030    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6" event={"ID":"2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe","Type":"ContainerDied","Data":"1d54e908f522dff7e77d2e957adbadbf50af605982a422a3ae81dd0f4535526b"}
Nov 03 01:26:00.182394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.182058    6000 scope.go:122] "RemoveContainer" containerID="27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2"
Nov 03 01:26:00.191558 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.191541    6000 scope.go:122] "RemoveContainer" containerID="27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2"
Nov 03 01:26:00.191930 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:26:00.191909    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2\": container with ID starting with 27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2 not found: ID does not exist" containerID="27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2"
Nov 03 01:26:00.191975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.191938    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2"} err="failed to get container status \"27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2\": rpc error: code = NotFound desc = could not find container \"27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2\": container with ID starting with 27a03c1eb1ec368f58706b26a4eec50e11dff38ad54db3baf42cf096d91d97d2 not found: ID does not exist"
Nov 03 01:26:00.228969 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.228942    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"]
Nov 03 01:26:00.239976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.239949    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-4m2w6"]
Nov 03 01:26:00.566315 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:00.566224    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" path="/var/lib/kubelet/pods/2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe/volumes"
Nov 03 01:26:08.174295 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:08.174237    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:26:10.188975 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.188933    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-846db64bd8-6tnsj"]
Nov 03 01:26:10.189655 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.189635    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" containerName="payload-pre-processing"
Nov 03 01:26:10.189709 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.189660    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" containerName="payload-pre-processing"
Nov 03 01:26:10.189803 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.189789    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d46ad2b-b643-4bb4-a01d-51c9c1f3c2fe" containerName="payload-pre-processing"
Nov 03 01:26:10.194526 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.194506    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.213337 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.213309    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-846db64bd8-6tnsj"]
Nov 03 01:26:10.253055 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.253028    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxqzv\" (UniqueName: \"kubernetes.io/projected/9949daa4-550e-4e71-bdb2-7d3f046eeb9d-kube-api-access-bxqzv\") pod \"payload-processing-846db64bd8-6tnsj\" (UID: \"9949daa4-550e-4e71-bdb2-7d3f046eeb9d\") " pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.253216 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.253179    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949daa4-550e-4e71-bdb2-7d3f046eeb9d-plugins-config-volume\") pod \"payload-processing-846db64bd8-6tnsj\" (UID: \"9949daa4-550e-4e71-bdb2-7d3f046eeb9d\") " pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.354156 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.354117    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949daa4-550e-4e71-bdb2-7d3f046eeb9d-plugins-config-volume\") pod \"payload-processing-846db64bd8-6tnsj\" (UID: \"9949daa4-550e-4e71-bdb2-7d3f046eeb9d\") " pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.354371 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.354209    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxqzv\" (UniqueName: \"kubernetes.io/projected/9949daa4-550e-4e71-bdb2-7d3f046eeb9d-kube-api-access-bxqzv\") pod \"payload-processing-846db64bd8-6tnsj\" (UID: \"9949daa4-550e-4e71-bdb2-7d3f046eeb9d\") " pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.354995 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.354973    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9949daa4-550e-4e71-bdb2-7d3f046eeb9d-plugins-config-volume\") pod \"payload-processing-846db64bd8-6tnsj\" (UID: \"9949daa4-550e-4e71-bdb2-7d3f046eeb9d\") " pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.361375 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.361345    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"]
Nov 03 01:26:10.365513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.365494    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.374141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.374110    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxqzv\" (UniqueName: \"kubernetes.io/projected/9949daa4-550e-4e71-bdb2-7d3f046eeb9d-kube-api-access-bxqzv\") pod \"payload-processing-846db64bd8-6tnsj\" (UID: \"9949daa4-550e-4e71-bdb2-7d3f046eeb9d\") " pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.383362 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.383337    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"]
Nov 03 01:26:10.455628 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.455543    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pnps\" (UniqueName: \"kubernetes.io/projected/0c0d40bf-fa16-4c0e-aa3c-595693ee1513-kube-api-access-7pnps\") pod \"payload-pre-processing-75b7db44c9-qzv2n\" (UID: \"0c0d40bf-fa16-4c0e-aa3c-595693ee1513\") " pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.455628 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.455604    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d40bf-fa16-4c0e-aa3c-595693ee1513-plugins-config-volume\") pod \"payload-pre-processing-75b7db44c9-qzv2n\" (UID: \"0c0d40bf-fa16-4c0e-aa3c-595693ee1513\") " pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.507742 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.507713    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:10.556392 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.556345    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pnps\" (UniqueName: \"kubernetes.io/projected/0c0d40bf-fa16-4c0e-aa3c-595693ee1513-kube-api-access-7pnps\") pod \"payload-pre-processing-75b7db44c9-qzv2n\" (UID: \"0c0d40bf-fa16-4c0e-aa3c-595693ee1513\") " pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.556582 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.556457    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d40bf-fa16-4c0e-aa3c-595693ee1513-plugins-config-volume\") pod \"payload-pre-processing-75b7db44c9-qzv2n\" (UID: \"0c0d40bf-fa16-4c0e-aa3c-595693ee1513\") " pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.557575 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.557544    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0c0d40bf-fa16-4c0e-aa3c-595693ee1513-plugins-config-volume\") pod \"payload-pre-processing-75b7db44c9-qzv2n\" (UID: \"0c0d40bf-fa16-4c0e-aa3c-595693ee1513\") " pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.581930 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.581901    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pnps\" (UniqueName: \"kubernetes.io/projected/0c0d40bf-fa16-4c0e-aa3c-595693ee1513-kube-api-access-7pnps\") pod \"payload-pre-processing-75b7db44c9-qzv2n\" (UID: \"0c0d40bf-fa16-4c0e-aa3c-595693ee1513\") " pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.690031 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.690002    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:10.799237 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.799195    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-846db64bd8-6tnsj"]
Nov 03 01:26:10.801569 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:26:10.801541    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9949daa4_550e_4e71_bdb2_7d3f046eeb9d.slice/crio-747d0cfe75650f91e81fa53ac96e8367ac4f5ae5929ba341623c61d4749c2b9a WatchSource:0}: Error finding container 747d0cfe75650f91e81fa53ac96e8367ac4f5ae5929ba341623c61d4749c2b9a: Status 404 returned error can't find the container with id 747d0cfe75650f91e81fa53ac96e8367ac4f5ae5929ba341623c61d4749c2b9a
Nov 03 01:26:10.980283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:10.980238    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"]
Nov 03 01:26:10.981686 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:26:10.981660    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c0d40bf_fa16_4c0e_aa3c_595693ee1513.slice/crio-089c8be2cffc5ed036360d9761ee09f94823497329effe69e0a05a33a3637fb6 WatchSource:0}: Error finding container 089c8be2cffc5ed036360d9761ee09f94823497329effe69e0a05a33a3637fb6: Status 404 returned error can't find the container with id 089c8be2cffc5ed036360d9761ee09f94823497329effe69e0a05a33a3637fb6
Nov 03 01:26:11.242453 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:11.242418    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n" event={"ID":"0c0d40bf-fa16-4c0e-aa3c-595693ee1513","Type":"ContainerStarted","Data":"8572a49e72b2d26fe40b02e8d7929e71b7f24068fc0fb306e5667736eaf54569"}
Nov 03 01:26:11.242922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:11.242471    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n" event={"ID":"0c0d40bf-fa16-4c0e-aa3c-595693ee1513","Type":"ContainerStarted","Data":"089c8be2cffc5ed036360d9761ee09f94823497329effe69e0a05a33a3637fb6"}
Nov 03 01:26:11.242922 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:11.242534    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:11.244073 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:11.244012    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-846db64bd8-6tnsj" event={"ID":"9949daa4-550e-4e71-bdb2-7d3f046eeb9d","Type":"ContainerStarted","Data":"2343fc7e392498f83498839738b00a09563d595dd7368d75ff9a5450e70fc32b"}
Nov 03 01:26:11.244073 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:11.244050    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-846db64bd8-6tnsj" event={"ID":"9949daa4-550e-4e71-bdb2-7d3f046eeb9d","Type":"ContainerStarted","Data":"747d0cfe75650f91e81fa53ac96e8367ac4f5ae5929ba341623c61d4749c2b9a"}
Nov 03 01:26:11.244073 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:11.244070    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:11.318416 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:11.318347    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n" podStartSLOduration=1.3183344209999999 podStartE2EDuration="1.318334421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:11.282713174 +0000 UTC m=+4808.838065198" watchObservedRunningTime="2026-11-03 01:26:11.318334421 +0000 UTC m=+4808.873686441"
Nov 03 01:26:19.894861 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:26:19.894823    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:26:22.252281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:22.252230    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-846db64bd8-6tnsj"
Nov 03 01:26:22.252691 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:22.252408    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-75b7db44c9-qzv2n"
Nov 03 01:26:22.289133 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:22.289083    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-846db64bd8-6tnsj" podStartSLOduration=12.289069401999999 podStartE2EDuration="12.289069402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:11.318062086 +0000 UTC m=+4808.873414134" watchObservedRunningTime="2026-11-03 01:26:22.289069402 +0000 UTC m=+4819.844421425"
Nov 03 01:26:22.311182 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:22.311148    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-b7f977d8-nps9f"]
Nov 03 01:26:22.347309 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:22.347278    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"]
Nov 03 01:26:25.573187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.573156    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"]
Nov 03 01:26:25.577165 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.577146    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.590931 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.590908    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Nov 03 01:26:25.602332 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.602304    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Nov 03 01:26:25.602506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.602368    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Nov 03 01:26:25.602506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.602444    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Nov 03 01:26:25.602506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.602459    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-82qg7\""
Nov 03 01:26:25.602506 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.602503    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"e2e-rotation-external-model-api-key\""
Nov 03 01:26:25.604735 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.604711    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"]
Nov 03 01:26:25.618504 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.618466    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.618688 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.618531    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jp55s\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-kube-api-access-jp55s\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.618761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.618745    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.618813 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.618790    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.719543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.719512    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.719749 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.719564    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.719749 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.719591    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.719749 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.719619    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jp55s\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-kube-api-access-jp55s\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.720410 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.720381    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-routing-overlay-volume\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.720806 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.720781    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-plugins-config-volume\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.722664 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.722638    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-provider-credentials\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.737822 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.737791    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jp55s\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-kube-api-access-jp55s\") pod \"payload-processing-external-model-5bcd7997-x6mgv\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") " pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:25.891673 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:25.891640    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:26.186094 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:26.186060    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"]
Nov 03 01:26:26.187177 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:26:26.187148    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d307893_1d5a_43aa_a657_7d3bf9ee9c9b.slice/crio-9f2759e79ff5698f7ffafe67c7e36ad9a592e1f55d513c65a34ab11d73fa7679 WatchSource:0}: Error finding container 9f2759e79ff5698f7ffafe67c7e36ad9a592e1f55d513c65a34ab11d73fa7679: Status 404 returned error can't find the container with id 9f2759e79ff5698f7ffafe67c7e36ad9a592e1f55d513c65a34ab11d73fa7679
Nov 03 01:26:26.315641 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:26.315608    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv" event={"ID":"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b","Type":"ContainerStarted","Data":"341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c"}
Nov 03 01:26:26.315834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:26.315663    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv" event={"ID":"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b","Type":"ContainerStarted","Data":"9f2759e79ff5698f7ffafe67c7e36ad9a592e1f55d513c65a34ab11d73fa7679"}
Nov 03 01:26:26.315834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:26.315811    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:26.353440 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:26.353389    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv" podStartSLOduration=1.353375805 podStartE2EDuration="1.353375805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:26:26.351363011 +0000 UTC m=+4823.906715036" watchObservedRunningTime="2026-11-03 01:26:26.353375805 +0000 UTC m=+4823.908727831"
Nov 03 01:26:26.719683 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:26.719648    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:26:27.312475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.312407    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-b7f977d8-nps9f" podUID="aed665f9-5ffc-438c-a98e-c52393835dc2" containerName="payload-processing" containerID="cri-o://367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7" gracePeriod=25
Nov 03 01:26:27.348224 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.348176    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb" podUID="562d41b3-f186-4131-a4b1-8ac819e24b1d" containerName="payload-pre-processing" containerID="cri-o://f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf" gracePeriod=25
Nov 03 01:26:27.682822 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.682501    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:26:27.714488 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.714457    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:26:27.755203 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.755166    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/562d41b3-f186-4131-a4b1-8ac819e24b1d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562d41b3-f186-4131-a4b1-8ac819e24b1d-plugins-config-volume\") pod \"562d41b3-f186-4131-a4b1-8ac819e24b1d\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") "
Nov 03 01:26:27.755730 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.755353    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed665f9-5ffc-438c-a98e-c52393835dc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed665f9-5ffc-438c-a98e-c52393835dc2-plugins-config-volume\") pod \"aed665f9-5ffc-438c-a98e-c52393835dc2\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") "
Nov 03 01:26:27.755730 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.755429    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/562d41b3-f186-4131-a4b1-8ac819e24b1d-kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/562d41b3-f186-4131-a4b1-8ac819e24b1d-kube-api-access-jqvds\") pod \"562d41b3-f186-4131-a4b1-8ac819e24b1d\" (UID: \"562d41b3-f186-4131-a4b1-8ac819e24b1d\") "
Nov 03 01:26:27.755730 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.755503    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed665f9-5ffc-438c-a98e-c52393835dc2-kube-api-access-jcb6g\" (UniqueName: \"kubernetes.io/projected/aed665f9-5ffc-438c-a98e-c52393835dc2-kube-api-access-jcb6g\") pod \"aed665f9-5ffc-438c-a98e-c52393835dc2\" (UID: \"aed665f9-5ffc-438c-a98e-c52393835dc2\") "
Nov 03 01:26:27.757673 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.757614    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/562d41b3-f186-4131-a4b1-8ac819e24b1d-plugins-config-volume" pod "562d41b3-f186-4131-a4b1-8ac819e24b1d" (UID: "562d41b3-f186-4131-a4b1-8ac819e24b1d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:26:27.758011 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.757965    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed665f9-5ffc-438c-a98e-c52393835dc2-plugins-config-volume" pod "aed665f9-5ffc-438c-a98e-c52393835dc2" (UID: "aed665f9-5ffc-438c-a98e-c52393835dc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:26:27.760196 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.760085    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/562d41b3-f186-4131-a4b1-8ac819e24b1d-kube-api-access-jqvds" pod "562d41b3-f186-4131-a4b1-8ac819e24b1d" (UID: "562d41b3-f186-4131-a4b1-8ac819e24b1d"). InnerVolumeSpecName "kube-api-access-jqvds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:26:27.760196 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.760162    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed665f9-5ffc-438c-a98e-c52393835dc2-kube-api-access-jcb6g" pod "aed665f9-5ffc-438c-a98e-c52393835dc2" (UID: "aed665f9-5ffc-438c-a98e-c52393835dc2"). InnerVolumeSpecName "kube-api-access-jcb6g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:26:27.856902 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.856800    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aed665f9-5ffc-438c-a98e-c52393835dc2-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:26:27.856902 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.856841    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqvds\" (UniqueName: \"kubernetes.io/projected/562d41b3-f186-4131-a4b1-8ac819e24b1d-kube-api-access-jqvds\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:26:27.856902 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.856870    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jcb6g\" (UniqueName: \"kubernetes.io/projected/aed665f9-5ffc-438c-a98e-c52393835dc2-kube-api-access-jcb6g\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:26:27.856902 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:27.856886    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/562d41b3-f186-4131-a4b1-8ac819e24b1d-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:26:28.374551 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.374511    6000 generic.go:356] "Generic (PLEG): container finished" podID="562d41b3-f186-4131-a4b1-8ac819e24b1d" containerID="f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf" exitCode=0
Nov 03 01:26:28.374768 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.374712    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"
Nov 03 01:26:28.375368 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.375341    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb" event={"ID":"562d41b3-f186-4131-a4b1-8ac819e24b1d","Type":"ContainerDied","Data":"f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf"}
Nov 03 01:26:28.375567 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.375547    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb" event={"ID":"562d41b3-f186-4131-a4b1-8ac819e24b1d","Type":"ContainerDied","Data":"4f986385a2d7229debbd109742af515cb32b5ef2d35f81671272cd60e34f0d8f"}
Nov 03 01:26:28.375711 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.375697    6000 scope.go:122] "RemoveContainer" containerID="f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf"
Nov 03 01:26:28.383718 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.383697    6000 generic.go:356] "Generic (PLEG): container finished" podID="aed665f9-5ffc-438c-a98e-c52393835dc2" containerID="367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7" exitCode=0
Nov 03 01:26:28.384347 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.384327    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b7f977d8-nps9f" event={"ID":"aed665f9-5ffc-438c-a98e-c52393835dc2","Type":"ContainerDied","Data":"367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7"}
Nov 03 01:26:28.384517 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.384501    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-b7f977d8-nps9f" event={"ID":"aed665f9-5ffc-438c-a98e-c52393835dc2","Type":"ContainerDied","Data":"183a53cc659b85b87c85b8528dd17a1c9c6f91428d719f3224be44c7950ecc22"}
Nov 03 01:26:28.384756 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.384741    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-b7f977d8-nps9f"
Nov 03 01:26:28.471294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.456296    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"]
Nov 03 01:26:28.471294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.456446    6000 scope.go:122] "RemoveContainer" containerID="f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf"
Nov 03 01:26:28.471294 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:26:28.467763    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf\": container with ID starting with f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf not found: ID does not exist" containerID="f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf"
Nov 03 01:26:28.471294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.467808    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf"} err="failed to get container status \"f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf\": rpc error: code = NotFound desc = could not find container \"f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf\": container with ID starting with f56cb3cb72805e0c8008d73e19110cbb3c55a46990377e5b5bd69f533177afdf not found: ID does not exist"
Nov 03 01:26:28.471294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.467836    6000 scope.go:122] "RemoveContainer" containerID="367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7"
Nov 03 01:26:28.471294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.467995    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-b5668cfcb-2xmwb"]
Nov 03 01:26:28.526290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.525411    6000 scope.go:122] "RemoveContainer" containerID="367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7"
Nov 03 01:26:28.526456 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:26:28.526288    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7\": container with ID starting with 367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7 not found: ID does not exist" containerID="367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7"
Nov 03 01:26:28.526456 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.526323    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7"} err="failed to get container status \"367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7\": rpc error: code = NotFound desc = could not find container \"367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7\": container with ID starting with 367c643518ba2b582fb53f37a14f8685a0e3da88c3bfeba86e33e185a519cde7 not found: ID does not exist"
Nov 03 01:26:28.774335 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.766997    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="562d41b3-f186-4131-a4b1-8ac819e24b1d" path="/var/lib/kubelet/pods/562d41b3-f186-4131-a4b1-8ac819e24b1d/volumes"
Nov 03 01:26:28.779416 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.779389    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-b7f977d8-nps9f"]
Nov 03 01:26:28.788292 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:28.787245    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-b7f977d8-nps9f"]
Nov 03 01:26:30.564980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:30.564947    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed665f9-5ffc-438c-a98e-c52393835dc2" path="/var/lib/kubelet/pods/aed665f9-5ffc-438c-a98e-c52393835dc2/volumes"
Nov 03 01:26:37.326010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:26:37.325975    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:26:49.895712 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:26:49.895672    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:27:19.894929 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:27:19.894841    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:27:49.895299 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:27:49.895239    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:28:02.448446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:02.448410    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"]
Nov 03 01:28:03.729232 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:03.729196    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:28:07.449336 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.449294    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv" podUID="1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" containerName="payload-processing" containerID="cri-o://341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c" gracePeriod=25
Nov 03 01:28:07.704029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.703967    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:28:07.857862 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.857831    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-provider-credentials\") pod \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") "
Nov 03 01:28:07.858054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.857917    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-kube-api-access-jp55s\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-kube-api-access-jp55s\") pod \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") "
Nov 03 01:28:07.858054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.858006    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-routing-overlay-volume\") pod \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") "
Nov 03 01:28:07.858054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.858031    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-plugins-config-volume\") pod \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\" (UID: \"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b\") "
Nov 03 01:28:07.858441 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.858415    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-routing-overlay-volume" pod "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" (UID: "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:07.858568 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.858476    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-plugins-config-volume" pod "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" (UID: "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:28:07.860092 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.860065    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-provider-credentials" pod "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" (UID: "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:07.860187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.860127    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-kube-api-access-jp55s" pod "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" (UID: "1d307893-1d5a-43aa-a657-7d3bf9ee9c9b"). InnerVolumeSpecName "kube-api-access-jp55s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:28:07.900486 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.900451    6000 generic.go:356] "Generic (PLEG): container finished" podID="1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" containerID="341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c" exitCode=0
Nov 03 01:28:07.900652 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.900501    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"
Nov 03 01:28:07.900652 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.900514    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv" event={"ID":"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b","Type":"ContainerDied","Data":"341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c"}
Nov 03 01:28:07.900652 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.900598    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv" event={"ID":"1d307893-1d5a-43aa-a657-7d3bf9ee9c9b","Type":"ContainerDied","Data":"9f2759e79ff5698f7ffafe67c7e36ad9a592e1f55d513c65a34ab11d73fa7679"}
Nov 03 01:28:07.900652 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.900628    6000 scope.go:122] "RemoveContainer" containerID="341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c"
Nov 03 01:28:07.910017 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.909995    6000 scope.go:122] "RemoveContainer" containerID="341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c"
Nov 03 01:28:07.910346 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:28:07.910319    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c\": container with ID starting with 341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c not found: ID does not exist" containerID="341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c"
Nov 03 01:28:07.910421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.910356    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c"} err="failed to get container status \"341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c\": rpc error: code = NotFound desc = could not find container \"341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c\": container with ID starting with 341bf40a99eb15acd519325ba9122ac60fda182122d16ab1b3341b1d8ffde71c not found: ID does not exist"
Nov 03 01:28:07.947537 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.947509    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"]
Nov 03 01:28:07.950456 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.950434    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-5bcd7997-x6mgv"]
Nov 03 01:28:07.959490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.959435    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jp55s\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-kube-api-access-jp55s\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:28:07.959490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.959455    6000 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-routing-overlay-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:28:07.959490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.959467    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:28:07.959490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:07.959476    6000 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b-provider-credentials\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:28:08.565547 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:08.565512    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" path="/var/lib/kubelet/pods/1d307893-1d5a-43aa-a657-7d3bf9ee9c9b/volumes"
Nov 03 01:28:10.539943 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:10.539910    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:28:19.895532 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:28:19.895492    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:28:26.319290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:26.319238    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:28:30.736481 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:30.736444    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:28:45.805289 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:28:45.805192    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:28:49.895076 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:28:49.895037    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:29:01.618141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:29:01.618103    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:29:19.895771 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:29:19.895728    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:29:24.128553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:29:24.128519    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:29:34.447551 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:29:34.447517    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:29:38.741746 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:29:38.741707    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:29:49.895343 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:29:49.895303    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:30:00.719631 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:00.719594    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:30:05.154193 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:05.154157    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:30:09.048817 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:09.048781    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:30:19.895480 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:30:19.895396    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:30:20.323418 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:20.323321    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:30:23.624480 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:23.624421    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:30:23.625175 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:23.625125    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:30:25.445634 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:25.445601    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:30:33.946695 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:33.946653    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:30:39.039111 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:39.039069    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:30:49.895519 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:30:49.895470    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:30:57.602832 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:30:57.602795    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:31:02.839547 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:31:02.839513    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:31:15.514209 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:31:15.514173    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:31:19.895174 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:31:19.895130    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:31:20.630338 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:31:20.630305    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:31:26.402603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:31:26.402567    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:31:44.596156 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:31:44.596078    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:31:49.894529 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:31:49.894489    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:31:55.041610 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:31:55.041574    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:32:11.935456 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:11.935421    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:32:15.508624 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.508592    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"]
Nov 03 01:32:15.509074 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509060    6000 cpu_manager.go:414] "containerMap: removing container" podUID="562d41b3-f186-4131-a4b1-8ac819e24b1d" containerName="payload-pre-processing"
Nov 03 01:32:15.509129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509076    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="562d41b3-f186-4131-a4b1-8ac819e24b1d" containerName="payload-pre-processing"
Nov 03 01:32:15.509129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509089    6000 cpu_manager.go:414] "containerMap: removing container" podUID="1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" containerName="payload-processing"
Nov 03 01:32:15.509129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509097    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" containerName="payload-processing"
Nov 03 01:32:15.509129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509117    6000 cpu_manager.go:414] "containerMap: removing container" podUID="aed665f9-5ffc-438c-a98e-c52393835dc2" containerName="payload-processing"
Nov 03 01:32:15.509129 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509123    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed665f9-5ffc-438c-a98e-c52393835dc2" containerName="payload-processing"
Nov 03 01:32:15.509348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509200    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed665f9-5ffc-438c-a98e-c52393835dc2" containerName="payload-processing"
Nov 03 01:32:15.509348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509214    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="562d41b3-f186-4131-a4b1-8ac819e24b1d" containerName="payload-pre-processing"
Nov 03 01:32:15.509348 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.509222    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d307893-1d5a-43aa-a657-7d3bf9ee9c9b" containerName="payload-processing"
Nov 03 01:32:15.512834 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.512812    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.527476 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.527442    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-462c10-openshift-default-dockercfg-ntht8\""
Nov 03 01:32:15.540198 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.540168    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"]
Nov 03 01:32:15.673344 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673308    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-socket\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673537 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673376    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-credential-socket\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673537 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673493    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-data\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673537    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-podinfo\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673561    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-certs\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673678 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673650    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a833f1-f2e7-488e-b974-28e63fd248f1-istiod-ca-cert\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673810 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673701    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljmvk\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-kube-api-access-ljmvk\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673810 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673766    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-envoy\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.673810 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.673803    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-token\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774387    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-credential-socket\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774487    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-data\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774532    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-podinfo\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774559    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-certs\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774591    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a833f1-f2e7-488e-b974-28e63fd248f1-istiod-ca-cert\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774634    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljmvk\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-kube-api-access-ljmvk\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774673    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-envoy\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774727    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-token\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774863    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-socket\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.774972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774907    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-credential-socket\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.775171 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.774995    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-certs\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.775226 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.775191    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-data\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.775299 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.775248    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-socket\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.775409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.775388    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a833f1-f2e7-488e-b974-28e63fd248f1-istiod-ca-cert\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.777097 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.777073    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-envoy\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.777347 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.777327    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-podinfo\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.792854 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.792822    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-token\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.792972 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.792909    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljmvk\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-kube-api-access-ljmvk\") pod \"e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") " pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:15.827062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:15.827038    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:16.136869 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.136838    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"]
Nov 03 01:32:16.138192 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:32:16.138167    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a833f1_f2e7_488e_b974_28e63fd248f1.slice/crio-e0b2086336040df8fbda7f4367a032f7ab99e6c0cff350a84c0e78a168deb7c4 WatchSource:0}: Error finding container e0b2086336040df8fbda7f4367a032f7ab99e6c0cff350a84c0e78a168deb7c4: Status 404 returned error can't find the container with id e0b2086336040df8fbda7f4367a032f7ab99e6c0cff350a84c0e78a168deb7c4
Nov 03 01:32:16.140429 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.140394    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:32:16.140553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.140472    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:32:16.140553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.140519    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:32:16.339910 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.339876    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" event={"ID":"32a833f1-f2e7-488e-b974-28e63fd248f1","Type":"ContainerStarted","Data":"923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b"}
Nov 03 01:32:16.340117 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.339927    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" event={"ID":"32a833f1-f2e7-488e-b974-28e63fd248f1","Type":"ContainerStarted","Data":"e0b2086336040df8fbda7f4367a032f7ab99e6c0cff350a84c0e78a168deb7c4"}
Nov 03 01:32:16.379760 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.379695    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podStartSLOduration=1.379680672 podStartE2EDuration="1.379680672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:16.377376252 +0000 UTC m=+5173.932728278" watchObservedRunningTime="2026-11-03 01:32:16.379680672 +0000 UTC m=+5173.935032695"
Nov 03 01:32:16.827555 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:16.827524    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:17.449650 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:17.449620    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:32:17.832837 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:17.832750    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:18.349776 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:18.349738    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:18.350988 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:18.350968    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:19.895444 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:32:19.895412    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:32:20.292326 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.292215    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"]
Nov 03 01:32:20.296239 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.296212    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.323489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.323463    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"]
Nov 03 01:32:20.338420 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.338386    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-462c10-serving-cert\""
Nov 03 01:32:20.418529 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.418496    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crngb\" (UniqueName: \"kubernetes.io/projected/9deac96d-f877-447e-9809-85e00c284287-kube-api-access-crngb\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.418726 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.418538    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.418726 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.418637    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9deac96d-f877-447e-9809-85e00c284287-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.418868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.418794    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-metrics-certs\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.520086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.520052    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crngb\" (UniqueName: \"kubernetes.io/projected/9deac96d-f877-447e-9809-85e00c284287-kube-api-access-crngb\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.520311 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.520100    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.520311 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.520143    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9deac96d-f877-447e-9809-85e00c284287-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.520524 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.520502    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-metrics-certs\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.521285 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.521243    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9deac96d-f877-447e-9809-85e00c284287-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.523029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.523002    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.523462 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.523440    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-metrics-certs\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.539732 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.539705    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crngb\" (UniqueName: \"kubernetes.io/projected/9deac96d-f877-447e-9809-85e00c284287-kube-api-access-crngb\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.610380 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.610354    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:20.624398 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.624368    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"]
Nov 03 01:32:20.628277 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.628226    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.645066 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.645023    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-462c10\""
Nov 03 01:32:20.646891 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.646853    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"]
Nov 03 01:32:20.664517 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.664456    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"]
Nov 03 01:32:20.669122 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.669090    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:20.686080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.686054    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e5394935-dockercfg-n4d44\""
Nov 03 01:32:20.690290 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.688451    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"]
Nov 03 01:32:20.824782 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.824740    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8dfc\" (UniqueName: \"kubernetes.io/projected/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-kube-api-access-s8dfc\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.825006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.824816    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.825006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.824915    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7rk\" (UniqueName: \"kubernetes.io/projected/9a4819de-85f0-49b0-88f8-2458a489747f-kube-api-access-cz7rk\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:20.825006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.824945    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4819de-85f0-49b0-88f8-2458a489747f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:20.919376 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.919347    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"]
Nov 03 01:32:20.920348 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:32:20.920322    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9deac96d_f877_447e_9809_85e00c284287.slice/crio-184953eb6d641ed9c9917422f63ada74383e1dde1fc85261b95b04109d85c78b WatchSource:0}: Error finding container 184953eb6d641ed9c9917422f63ada74383e1dde1fc85261b95b04109d85c78b: Status 404 returned error can't find the container with id 184953eb6d641ed9c9917422f63ada74383e1dde1fc85261b95b04109d85c78b
Nov 03 01:32:20.921839 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.921821    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:32:20.926135 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.926102    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.926234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.926165    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7rk\" (UniqueName: \"kubernetes.io/projected/9a4819de-85f0-49b0-88f8-2458a489747f-kube-api-access-cz7rk\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:20.926234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.926191    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4819de-85f0-49b0-88f8-2458a489747f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:20.926390 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.926310    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8dfc\" (UniqueName: \"kubernetes.io/projected/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-kube-api-access-s8dfc\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.926894 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.926863    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.926990 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.926942    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4819de-85f0-49b0-88f8-2458a489747f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:20.944227 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.944207    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8dfc\" (UniqueName: \"kubernetes.io/projected/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-kube-api-access-s8dfc\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.944340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.944234    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7rk\" (UniqueName: \"kubernetes.io/projected/9a4819de-85f0-49b0-88f8-2458a489747f-kube-api-access-cz7rk\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:20.952098 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.952077    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:20.994846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:20.994760    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:21.245241 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.245216    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"]
Nov 03 01:32:21.246883 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:32:21.246843    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae0abedc_5bb9_46d6_8fe8_a22e620ed877.slice/crio-dc4465ce16573c8e3e9400339dfac0368a98bd86a1364009fe7c8c9e1d6f0d61 WatchSource:0}: Error finding container dc4465ce16573c8e3e9400339dfac0368a98bd86a1364009fe7c8c9e1d6f0d61: Status 404 returned error can't find the container with id dc4465ce16573c8e3e9400339dfac0368a98bd86a1364009fe7c8c9e1d6f0d61
Nov 03 01:32:21.300489 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.300461    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"]
Nov 03 01:32:21.301543 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:32:21.301521    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a4819de_85f0_49b0_88f8_2458a489747f.slice/crio-0a6c90d6cc779e842c1152bffaad9b1e1b4a241466e9e2597c32e01adcbd6b9e WatchSource:0}: Error finding container 0a6c90d6cc779e842c1152bffaad9b1e1b4a241466e9e2597c32e01adcbd6b9e: Status 404 returned error can't find the container with id 0a6c90d6cc779e842c1152bffaad9b1e1b4a241466e9e2597c32e01adcbd6b9e
Nov 03 01:32:21.366543 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.366510    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" event={"ID":"9a4819de-85f0-49b0-88f8-2458a489747f","Type":"ContainerStarted","Data":"0a6c90d6cc779e842c1152bffaad9b1e1b4a241466e9e2597c32e01adcbd6b9e"}
Nov 03 01:32:21.368151 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.368115    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" event={"ID":"ae0abedc-5bb9-46d6-8fe8-a22e620ed877","Type":"ContainerStarted","Data":"ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d"}
Nov 03 01:32:21.368318 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.368295    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:21.368394 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.368337    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" event={"ID":"ae0abedc-5bb9-46d6-8fe8-a22e620ed877","Type":"ContainerStarted","Data":"dc4465ce16573c8e3e9400339dfac0368a98bd86a1364009fe7c8c9e1d6f0d61"}
Nov 03 01:32:21.369472 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.369450    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd" event={"ID":"9deac96d-f877-447e-9809-85e00c284287","Type":"ContainerStarted","Data":"184953eb6d641ed9c9917422f63ada74383e1dde1fc85261b95b04109d85c78b"}
Nov 03 01:32:21.403783 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:21.403737    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" podStartSLOduration=1.403725254 podStartE2EDuration="1.403725254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:21.403338471 +0000 UTC m=+5178.958690496" watchObservedRunningTime="2026-11-03 01:32:21.403725254 +0000 UTC m=+5178.959077278"
Nov 03 01:32:22.376153 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:22.376101    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" event={"ID":"9a4819de-85f0-49b0-88f8-2458a489747f","Type":"ContainerStarted","Data":"8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9"}
Nov 03 01:32:22.376674 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:22.376200    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:22.377634 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:22.377606    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd" event={"ID":"9deac96d-f877-447e-9809-85e00c284287","Type":"ContainerStarted","Data":"be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e"}
Nov 03 01:32:22.377815 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:22.377799    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:22.413859 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:22.413803    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" podStartSLOduration=2.413787244 podStartE2EDuration="2.413787244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:22.412305664 +0000 UTC m=+5179.967657688" watchObservedRunningTime="2026-11-03 01:32:22.413787244 +0000 UTC m=+5179.969139270"
Nov 03 01:32:22.449104 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:22.449057    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd" podStartSLOduration=1.938861478 podStartE2EDuration="2.449042686s" totalImagesPullingTime="510.181208ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:22.448720756 +0000 UTC m=+5180.004072771" watchObservedRunningTime="2026-11-03 01:32:22.449042686 +0000 UTC m=+5180.004394734"
Nov 03 01:32:23.136281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:23.136216    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:32:28.411282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:28.409424    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:32:32.381351 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:32.381321    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:32:33.350921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.350884    6000 patch_prober.go:29] interesting pod/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body=
Nov 03 01:32:33.351150 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.350948    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused"
Nov 03 01:32:33.385842 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.385803    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:32:33.457248 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.457216    6000 generic.go:356] "Generic (PLEG): container finished" podID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerID="923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b" exitCode=0
Nov 03 01:32:33.457447 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.457298    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" event={"ID":"32a833f1-f2e7-488e-b974-28e63fd248f1","Type":"ContainerDied","Data":"923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b"}
Nov 03 01:32:33.457758 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.457741    6000 scope.go:122] "RemoveContainer" containerID="923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b"
Nov 03 01:32:33.459168 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.459129    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:32:33.459300 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.459223    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:32:33.459356 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:33.459306    6000 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892004Ki","pods":"250"}
Nov 03 01:32:34.465319 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:34.465284    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" event={"ID":"32a833f1-f2e7-488e-b974-28e63fd248f1","Type":"ContainerStarted","Data":"0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a"}
Nov 03 01:32:34.827601 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:34.827518    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:34.827777 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:34.827736    6000 patch_prober.go:29] interesting pod/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body=
Nov 03 01:32:34.827821 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:34.827784    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused"
Nov 03 01:32:35.412980 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.412513    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"]
Nov 03 01:32:35.416498 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.416217    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.431001 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.430970    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"]
Nov 03 01:32:35.438791 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.438761    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-462c10\"/\"openshift-service-ca.crt\""
Nov 03 01:32:35.438791 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.438768    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-462c10\"/\"kube-root-ca.crt\""
Nov 03 01:32:35.512512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.512475    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-462c10\"/\"simulator-single-main-462c10-kserve-self-signed-certs\""
Nov 03 01:32:35.513070 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.512580    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-462c10\"/\"default-dockercfg-nc9tp\""
Nov 03 01:32:35.519180 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.519140    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-model-cache\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.519379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.519197    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-home\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.519379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.519287    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tmp-dir\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.519379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.519345    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvn7q\" (UniqueName: \"kubernetes.io/projected/e1787e83-d6f6-4207-bc89-3aa39bed59aa-kube-api-access-tvn7q\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.519564 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.519451    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tls-certs\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.519564 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.519490    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-dshm\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.620605 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.620550    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-model-cache\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.620831 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.620631    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-home\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.620831 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.620781    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tmp-dir\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.620971 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.620891    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvn7q\" (UniqueName: \"kubernetes.io/projected/e1787e83-d6f6-4207-bc89-3aa39bed59aa-kube-api-access-tvn7q\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.621029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.620972    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tls-certs\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.621029 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.621011    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-dshm\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.621147 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.621055    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-model-cache\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.621147 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.621114    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-home\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.621296 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.621146    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tmp-dir\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.623505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.623481    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-dshm\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.623819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.623798    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tls-certs\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.650375 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.645127    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvn7q\" (UniqueName: \"kubernetes.io/projected/e1787e83-d6f6-4207-bc89-3aa39bed59aa-kube-api-access-tvn7q\") pod \"simulator-single-main-462c10-kserve-686c7cb554-2zkqr\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") " pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.733114 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.733026    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:35.827457 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.827422    6000 patch_prober.go:29] interesting pod/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body=
Nov 03 01:32:35.827607 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:35.827483    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused"
Nov 03 01:32:36.049619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:36.049587    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"]
Nov 03 01:32:36.050810 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:32:36.050779    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1787e83_d6f6_4207_bc89_3aa39bed59aa.slice/crio-4fd9f2912a263d25dd2306803aed1d5f946c0056119558d4731f7c81f1498c53 WatchSource:0}: Error finding container 4fd9f2912a263d25dd2306803aed1d5f946c0056119558d4731f7c81f1498c53: Status 404 returned error can't find the container with id 4fd9f2912a263d25dd2306803aed1d5f946c0056119558d4731f7c81f1498c53
Nov 03 01:32:36.480409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:36.480369    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr" event={"ID":"e1787e83-d6f6-4207-bc89-3aa39bed59aa","Type":"ContainerStarted","Data":"845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3"}
Nov 03 01:32:36.480621 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:36.480422    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr" event={"ID":"e1787e83-d6f6-4207-bc89-3aa39bed59aa","Type":"ContainerStarted","Data":"4fd9f2912a263d25dd2306803aed1d5f946c0056119558d4731f7c81f1498c53"}
Nov 03 01:32:36.555982 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:36.555914    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr" podStartSLOduration=1.555897686 podStartE2EDuration="1.555897686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:32:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:32:36.553666576 +0000 UTC m=+5194.109018599" watchObservedRunningTime="2026-11-03 01:32:36.555897686 +0000 UTC m=+5194.111249709"
Nov 03 01:32:36.733329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:36.733181    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:36.828269 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:36.828226    6000 patch_prober.go:29] interesting pod/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body=
Nov 03 01:32:36.828460 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:36.828311    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused"
Nov 03 01:32:37.609283 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:37.608629    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:32:37.828019 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:37.827979    6000 patch_prober.go:29] interesting pod/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body=
Nov 03 01:32:37.828223 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:37.828055    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused"
Nov 03 01:32:38.827619 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:38.827584    6000 patch_prober.go:29] interesting pod/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused" start-of-body=
Nov 03 01:32:38.828095 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:38.827641    6000 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.71:15021/healthz/ready\": dial tcp 10.134.1.71:15021: connect: connection refused"
Nov 03 01:32:39.948086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:39.948054    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:39.948665 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:39.948404    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:39.949094 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:39.949076    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:32:45.733211 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:45.733176    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:45.746051 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:45.746027    6000 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:46.567655 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:46.567626    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:32:49.895482 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:32:49.895447    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:32:54.028512 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:32:54.028462    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:33:18.123883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:18.123796    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:33:19.895725 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:33:19.895685    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:33:21.851917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:21.851882    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"]
Nov 03 01:33:23.104814 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:23.104777    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"]
Nov 03 01:33:23.131795 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:23.131757    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"]
Nov 03 01:33:25.783583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:25.783549    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"]
Nov 03 01:33:25.784093 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:25.783750    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd" podUID="9deac96d-f877-447e-9809-85e00c284287" containerName="maas-api" containerID="cri-o://be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e" gracePeriod=30
Nov 03 01:33:26.250050 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.250024    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:33:26.379705 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.379673    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9deac96d-f877-447e-9809-85e00c284287-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9deac96d-f877-447e-9809-85e00c284287-openshift-service-ca\") pod \"9deac96d-f877-447e-9809-85e00c284287\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") "
Nov 03 01:33:26.379885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.379712    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-maas-api-tls\") pod \"9deac96d-f877-447e-9809-85e00c284287\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") "
Nov 03 01:33:26.379885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.379740    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9deac96d-f877-447e-9809-85e00c284287-kube-api-access-crngb\" (UniqueName: \"kubernetes.io/projected/9deac96d-f877-447e-9809-85e00c284287-kube-api-access-crngb\") pod \"9deac96d-f877-447e-9809-85e00c284287\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") "
Nov 03 01:33:26.379885 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.379859    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-metrics-certs\") pod \"9deac96d-f877-447e-9809-85e00c284287\" (UID: \"9deac96d-f877-447e-9809-85e00c284287\") "
Nov 03 01:33:26.380176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.380146    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9deac96d-f877-447e-9809-85e00c284287-openshift-service-ca" pod "9deac96d-f877-447e-9809-85e00c284287" (UID: "9deac96d-f877-447e-9809-85e00c284287"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:26.380444 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.380418    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9deac96d-f877-447e-9809-85e00c284287-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:26.382163 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.382138    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-metrics-certs" pod "9deac96d-f877-447e-9809-85e00c284287" (UID: "9deac96d-f877-447e-9809-85e00c284287"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:26.382287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.382162    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-maas-api-tls" pod "9deac96d-f877-447e-9809-85e00c284287" (UID: "9deac96d-f877-447e-9809-85e00c284287"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:26.382287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.382175    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9deac96d-f877-447e-9809-85e00c284287-kube-api-access-crngb" pod "9deac96d-f877-447e-9809-85e00c284287" (UID: "9deac96d-f877-447e-9809-85e00c284287"). InnerVolumeSpecName "kube-api-access-crngb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:26.481785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.481748    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:26.481785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.481779    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crngb\" (UniqueName: \"kubernetes.io/projected/9deac96d-f877-447e-9809-85e00c284287-kube-api-access-crngb\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:26.481785 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.481789    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9deac96d-f877-447e-9809-85e00c284287-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:26.756792 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.756695    6000 generic.go:356] "Generic (PLEG): container finished" podID="9deac96d-f877-447e-9809-85e00c284287" containerID="be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e" exitCode=0
Nov 03 01:33:26.756792 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.756746    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"
Nov 03 01:33:26.756792 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.756774    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd" event={"ID":"9deac96d-f877-447e-9809-85e00c284287","Type":"ContainerDied","Data":"be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e"}
Nov 03 01:33:26.757093 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.756825    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd" event={"ID":"9deac96d-f877-447e-9809-85e00c284287","Type":"ContainerDied","Data":"184953eb6d641ed9c9917422f63ada74383e1dde1fc85261b95b04109d85c78b"}
Nov 03 01:33:26.757093 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.756849    6000 scope.go:122] "RemoveContainer" containerID="be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e"
Nov 03 01:33:26.766128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.766105    6000 scope.go:122] "RemoveContainer" containerID="be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e"
Nov 03 01:33:26.766424 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:33:26.766401    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e\": container with ID starting with be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e not found: ID does not exist" containerID="be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e"
Nov 03 01:33:26.766493 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.766433    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e"} err="failed to get container status \"be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e\": rpc error: code = NotFound desc = could not find container \"be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e\": container with ID starting with be3c3a8a7b4e76a50f94bf8c1d00861046b5eab595f2f12d3071ee7acb87fc6e not found: ID does not exist"
Nov 03 01:33:26.803948 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.803915    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"]
Nov 03 01:33:26.816288 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:26.816241    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-4k6jd"]
Nov 03 01:33:27.946127 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:27.946092    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:33:28.105208 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:28.105161    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" podUID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" containerName="payload-processing" containerID="cri-o://ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d" gracePeriod=25
Nov 03 01:33:28.132811 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:28.132746    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" podUID="9a4819de-85f0-49b0-88f8-2458a489747f" containerName="payload-pre-processing" containerID="cri-o://8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9" gracePeriod=25
Nov 03 01:33:28.565988 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:28.565946    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9deac96d-f877-447e-9809-85e00c284287" path="/var/lib/kubelet/pods/9deac96d-f877-447e-9809-85e00c284287/volumes"
Nov 03 01:33:32.379062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:32.379024    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" podUID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:33:33.384203 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:33.384164    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" podUID="9a4819de-85f0-49b0-88f8-2458a489747f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:33:36.873913 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:36.873852    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr" podUID="e1787e83-d6f6-4207-bc89-3aa39bed59aa" containerName="main" containerID="cri-o://845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3" gracePeriod=45
Nov 03 01:33:37.133720 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.133659    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:33:37.194638 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.194606    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-home\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-home\") pod \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") "
Nov 03 01:33:37.194846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.194661    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-model-cache\") pod \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") "
Nov 03 01:33:37.194846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.194767    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1787e83-d6f6-4207-bc89-3aa39bed59aa-kube-api-access-tvn7q\" (UniqueName: \"kubernetes.io/projected/e1787e83-d6f6-4207-bc89-3aa39bed59aa-kube-api-access-tvn7q\") pod \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") "
Nov 03 01:33:37.194846 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.194829    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-dshm\") pod \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") "
Nov 03 01:33:37.195061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.194897    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tls-certs\" (UniqueName: \"kubernetes.io/secret/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tls-certs\") pod \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") "
Nov 03 01:33:37.195061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.194899    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-home" pod "e1787e83-d6f6-4207-bc89-3aa39bed59aa" (UID: "e1787e83-d6f6-4207-bc89-3aa39bed59aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:37.195061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.195003    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tmp-dir\") pod \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\" (UID: \"e1787e83-d6f6-4207-bc89-3aa39bed59aa\") "
Nov 03 01:33:37.195061 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.195031    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-model-cache" pod "e1787e83-d6f6-4207-bc89-3aa39bed59aa" (UID: "e1787e83-d6f6-4207-bc89-3aa39bed59aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:37.195446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.195405    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tmp-dir" pod "e1787e83-d6f6-4207-bc89-3aa39bed59aa" (UID: "e1787e83-d6f6-4207-bc89-3aa39bed59aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:37.195612 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.195485    6000 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-home\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:37.195612 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.195507    6000 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-model-cache\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:37.197594 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.197551    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-dshm" pod "e1787e83-d6f6-4207-bc89-3aa39bed59aa" (UID: "e1787e83-d6f6-4207-bc89-3aa39bed59aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:33:37.197719 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.197679    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1787e83-d6f6-4207-bc89-3aa39bed59aa-kube-api-access-tvn7q" pod "e1787e83-d6f6-4207-bc89-3aa39bed59aa" (UID: "e1787e83-d6f6-4207-bc89-3aa39bed59aa"). InnerVolumeSpecName "kube-api-access-tvn7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:37.197779 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.197738    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tls-certs" pod "e1787e83-d6f6-4207-bc89-3aa39bed59aa" (UID: "e1787e83-d6f6-4207-bc89-3aa39bed59aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:33:37.296801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.296759    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvn7q\" (UniqueName: \"kubernetes.io/projected/e1787e83-d6f6-4207-bc89-3aa39bed59aa-kube-api-access-tvn7q\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:37.296801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.296793    6000 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-dshm\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:37.296801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.296805    6000 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tls-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:37.296801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.296813    6000 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1787e83-d6f6-4207-bc89-3aa39bed59aa-tmp-dir\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:37.818448 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.818415    6000 generic.go:356] "Generic (PLEG): container finished" podID="e1787e83-d6f6-4207-bc89-3aa39bed59aa" containerID="845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3" exitCode=0
Nov 03 01:33:37.818629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.818464    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"
Nov 03 01:33:37.818629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.818497    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr" event={"ID":"e1787e83-d6f6-4207-bc89-3aa39bed59aa","Type":"ContainerDied","Data":"845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3"}
Nov 03 01:33:37.818629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.818542    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr" event={"ID":"e1787e83-d6f6-4207-bc89-3aa39bed59aa","Type":"ContainerDied","Data":"4fd9f2912a263d25dd2306803aed1d5f946c0056119558d4731f7c81f1498c53"}
Nov 03 01:33:37.818629 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.818565    6000 scope.go:122] "RemoveContainer" containerID="845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3"
Nov 03 01:33:37.828178 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.828133    6000 scope.go:122] "RemoveContainer" containerID="845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3"
Nov 03 01:33:37.828458 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:33:37.828438    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3\": container with ID starting with 845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3 not found: ID does not exist" containerID="845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3"
Nov 03 01:33:37.828502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.828466    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3"} err="failed to get container status \"845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3\": rpc error: code = NotFound desc = could not find container \"845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3\": container with ID starting with 845f9530997e79f49dc13b8710b7bc2794666358d3cc68c2b1425f04410398e3 not found: ID does not exist"
Nov 03 01:33:37.867810 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.867776    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"]
Nov 03 01:33:37.878819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:37.878773    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-462c10/simulator-single-main-462c10-kserve-686c7cb554-2zkqr"]
Nov 03 01:33:38.566310 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:38.566245    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1787e83-d6f6-4207-bc89-3aa39bed59aa" path="/var/lib/kubelet/pods/e1787e83-d6f6-4207-bc89-3aa39bed59aa/volumes"
Nov 03 01:33:42.379610 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:42.379569    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" podUID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:33:43.383790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:43.383752    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" podUID="9a4819de-85f0-49b0-88f8-2458a489747f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 03 01:33:48.384533 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.384506    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:33:48.406010 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.405985    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:33:48.509750 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.509712    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-kube-api-access-s8dfc\" (UniqueName: \"kubernetes.io/projected/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-kube-api-access-s8dfc\") pod \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") "
Nov 03 01:33:48.509949 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.509815    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-plugins-config-volume\") pod \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\" (UID: \"ae0abedc-5bb9-46d6-8fe8-a22e620ed877\") "
Nov 03 01:33:48.509949 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.509910    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a4819de-85f0-49b0-88f8-2458a489747f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4819de-85f0-49b0-88f8-2458a489747f-plugins-config-volume\") pod \"9a4819de-85f0-49b0-88f8-2458a489747f\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") "
Nov 03 01:33:48.510081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.509962    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a4819de-85f0-49b0-88f8-2458a489747f-kube-api-access-cz7rk\" (UniqueName: \"kubernetes.io/projected/9a4819de-85f0-49b0-88f8-2458a489747f-kube-api-access-cz7rk\") pod \"9a4819de-85f0-49b0-88f8-2458a489747f\" (UID: \"9a4819de-85f0-49b0-88f8-2458a489747f\") "
Nov 03 01:33:48.510296 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.510238    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-plugins-config-volume" pod "ae0abedc-5bb9-46d6-8fe8-a22e620ed877" (UID: "ae0abedc-5bb9-46d6-8fe8-a22e620ed877"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:48.510377 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.510271    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a4819de-85f0-49b0-88f8-2458a489747f-plugins-config-volume" pod "9a4819de-85f0-49b0-88f8-2458a489747f" (UID: "9a4819de-85f0-49b0-88f8-2458a489747f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:33:48.512120 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.512099    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-kube-api-access-s8dfc" pod "ae0abedc-5bb9-46d6-8fe8-a22e620ed877" (UID: "ae0abedc-5bb9-46d6-8fe8-a22e620ed877"). InnerVolumeSpecName "kube-api-access-s8dfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:48.512234 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.512215    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a4819de-85f0-49b0-88f8-2458a489747f-kube-api-access-cz7rk" pod "9a4819de-85f0-49b0-88f8-2458a489747f" (UID: "9a4819de-85f0-49b0-88f8-2458a489747f"). InnerVolumeSpecName "kube-api-access-cz7rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:33:48.611741 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.611667    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:48.611741 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.611692    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a4819de-85f0-49b0-88f8-2458a489747f-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:48.611741 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.611701    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz7rk\" (UniqueName: \"kubernetes.io/projected/9a4819de-85f0-49b0-88f8-2458a489747f-kube-api-access-cz7rk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:48.611741 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.611710    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8dfc\" (UniqueName: \"kubernetes.io/projected/ae0abedc-5bb9-46d6-8fe8-a22e620ed877-kube-api-access-s8dfc\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:33:48.874452 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.874360    6000 generic.go:356] "Generic (PLEG): container finished" podID="9a4819de-85f0-49b0-88f8-2458a489747f" containerID="8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9" exitCode=0
Nov 03 01:33:48.874452 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.874411    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"
Nov 03 01:33:48.874452 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.874441    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" event={"ID":"9a4819de-85f0-49b0-88f8-2458a489747f","Type":"ContainerDied","Data":"8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9"}
Nov 03 01:33:48.874761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.874494    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b" event={"ID":"9a4819de-85f0-49b0-88f8-2458a489747f","Type":"ContainerDied","Data":"0a6c90d6cc779e842c1152bffaad9b1e1b4a241466e9e2597c32e01adcbd6b9e"}
Nov 03 01:33:48.874761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.874519    6000 scope.go:122] "RemoveContainer" containerID="8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9"
Nov 03 01:33:48.875888 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.875864    6000 generic.go:356] "Generic (PLEG): container finished" podID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" containerID="ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d" exitCode=0
Nov 03 01:33:48.876006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.875894    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" event={"ID":"ae0abedc-5bb9-46d6-8fe8-a22e620ed877","Type":"ContainerDied","Data":"ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d"}
Nov 03 01:33:48.876006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.875918    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"
Nov 03 01:33:48.876006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.875926    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25" event={"ID":"ae0abedc-5bb9-46d6-8fe8-a22e620ed877","Type":"ContainerDied","Data":"dc4465ce16573c8e3e9400339dfac0368a98bd86a1364009fe7c8c9e1d6f0d61"}
Nov 03 01:33:48.883326 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.883307    6000 scope.go:122] "RemoveContainer" containerID="8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9"
Nov 03 01:33:48.883560 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:33:48.883543    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9\": container with ID starting with 8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9 not found: ID does not exist" containerID="8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9"
Nov 03 01:33:48.883606 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.883567    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9"} err="failed to get container status \"8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9\": rpc error: code = NotFound desc = could not find container \"8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9\": container with ID starting with 8ca902c9ce8395efc573831ea69aae52714f329edad96ae3b6c74295e35cbdb9 not found: ID does not exist"
Nov 03 01:33:48.883606 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.883583    6000 scope.go:122] "RemoveContainer" containerID="ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d"
Nov 03 01:33:48.891604 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.891587    6000 scope.go:122] "RemoveContainer" containerID="ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d"
Nov 03 01:33:48.891862 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:33:48.891843    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d\": container with ID starting with ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d not found: ID does not exist" containerID="ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d"
Nov 03 01:33:48.891941 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.891867    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d"} err="failed to get container status \"ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d\": rpc error: code = NotFound desc = could not find container \"ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d\": container with ID starting with ba2f72a3bc2d4bf537bc76812fa9c62ca6306689ba2985116aadf43acc29179d not found: ID does not exist"
Nov 03 01:33:48.921743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.921717    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"]
Nov 03 01:33:48.932728 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.932704    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dtss5b"]
Nov 03 01:33:48.956776 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.956746    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"]
Nov 03 01:33:48.958082 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:48.958059    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fn8g25"]
Nov 03 01:33:49.895716 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:33:49.895683    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:33:50.565495 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:50.565462    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a4819de-85f0-49b0-88f8-2458a489747f" path="/var/lib/kubelet/pods/9a4819de-85f0-49b0-88f8-2458a489747f/volumes"
Nov 03 01:33:50.565981 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:50.565964    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" path="/var/lib/kubelet/pods/ae0abedc-5bb9-46d6-8fe8-a22e620ed877/volumes"
Nov 03 01:33:57.919635 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.919602    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"]
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920033    6000 cpu_manager.go:414] "containerMap: removing container" podUID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" containerName="payload-processing"
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920044    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" containerName="payload-processing"
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920062    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9deac96d-f877-447e-9809-85e00c284287" containerName="maas-api"
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920067    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9deac96d-f877-447e-9809-85e00c284287" containerName="maas-api"
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920079    6000 cpu_manager.go:414] "containerMap: removing container" podUID="9a4819de-85f0-49b0-88f8-2458a489747f" containerName="payload-pre-processing"
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920085    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a4819de-85f0-49b0-88f8-2458a489747f" containerName="payload-pre-processing"
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920091    6000 cpu_manager.go:414] "containerMap: removing container" podUID="e1787e83-d6f6-4207-bc89-3aa39bed59aa" containerName="main"
Nov 03 01:33:57.920148 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920095    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1787e83-d6f6-4207-bc89-3aa39bed59aa" containerName="main"
Nov 03 01:33:57.920463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920181    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae0abedc-5bb9-46d6-8fe8-a22e620ed877" containerName="payload-processing"
Nov 03 01:33:57.920463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920190    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9deac96d-f877-447e-9809-85e00c284287" containerName="maas-api"
Nov 03 01:33:57.920463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920197    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a4819de-85f0-49b0-88f8-2458a489747f" containerName="payload-pre-processing"
Nov 03 01:33:57.920463 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.920204    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1787e83-d6f6-4207-bc89-3aa39bed59aa" containerName="main"
Nov 03 01:33:57.925080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.925057    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:57.964855 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.964816    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-462c10-serving-cert\""
Nov 03 01:33:57.967034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:57.966992    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"]
Nov 03 01:33:58.002693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.002661    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.002858 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.002792    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2gcg\" (UniqueName: \"kubernetes.io/projected/2a02b1db-df08-4339-a68a-6b92094afde3-kube-api-access-r2gcg\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.002858 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.002839    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a02b1db-df08-4339-a68a-6b92094afde3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.002952 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.002930    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.104314 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.104276    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.104519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.104370    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.104519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.104424    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2gcg\" (UniqueName: \"kubernetes.io/projected/2a02b1db-df08-4339-a68a-6b92094afde3-kube-api-access-r2gcg\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.104519 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.104451    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a02b1db-df08-4339-a68a-6b92094afde3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.105142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.105091    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a02b1db-df08-4339-a68a-6b92094afde3-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.107146 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.107114    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-metrics-certs\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.107305 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.107114    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.126475 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.126445    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2gcg\" (UniqueName: \"kubernetes.io/projected/2a02b1db-df08-4339-a68a-6b92094afde3-kube-api-access-r2gcg\") pod \"maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.237671 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.237557    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:58.384788 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.384751    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"]
Nov 03 01:33:58.388722 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.388705    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.403984 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.403963    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-462c10\""
Nov 03 01:33:58.411743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.411721    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"]
Nov 03 01:33:58.425329 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.425308    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-e5394935-dockercfg-vl2sz\""
Nov 03 01:33:58.487110 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.487061    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"]
Nov 03 01:33:58.495080 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.495004    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.512770 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.511759    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5f35fd-4043-4391-98ac-2404de3c59b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.512770 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.512083    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7b228\" (UniqueName: \"kubernetes.io/projected/7e5f35fd-4043-4391-98ac-2404de3c59b6-kube-api-access-7b228\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.522128 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.522095    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"]
Nov 03 01:33:58.547868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.547840    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"]
Nov 03 01:33:58.553081 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:33:58.553050    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a02b1db_df08_4339_a68a_6b92094afde3.slice/crio-88d53e143912f08648b448942bab2c8afbe740165a72cda6cdf52b53a9cca493 WatchSource:0}: Error finding container 88d53e143912f08648b448942bab2c8afbe740165a72cda6cdf52b53a9cca493: Status 404 returned error can't find the container with id 88d53e143912f08648b448942bab2c8afbe740165a72cda6cdf52b53a9cca493
Nov 03 01:33:58.613227 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.613191    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7b228\" (UniqueName: \"kubernetes.io/projected/7e5f35fd-4043-4391-98ac-2404de3c59b6-kube-api-access-7b228\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.613446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.613279    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.613446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.613386    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5f35fd-4043-4391-98ac-2404de3c59b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.613446 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.613441    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwbxp\" (UniqueName: \"kubernetes.io/projected/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-kube-api-access-mwbxp\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.614181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.614160    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5f35fd-4043-4391-98ac-2404de3c59b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.632528 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.632489    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7b228\" (UniqueName: \"kubernetes.io/projected/7e5f35fd-4043-4391-98ac-2404de3c59b6-kube-api-access-7b228\") pod \"payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.702708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.702675    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:58.714373 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.714349    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.714497 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.714437    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwbxp\" (UniqueName: \"kubernetes.io/projected/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-kube-api-access-mwbxp\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.715081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.715055    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.733801 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.733774    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwbxp\" (UniqueName: \"kubernetes.io/projected/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-kube-api-access-mwbxp\") pod \"payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.830917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.830862    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:58.926367 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.926333    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np" event={"ID":"2a02b1db-df08-4339-a68a-6b92094afde3","Type":"ContainerStarted","Data":"88d53e143912f08648b448942bab2c8afbe740165a72cda6cdf52b53a9cca493"}
Nov 03 01:33:58.997285 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:58.997236    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"]
Nov 03 01:33:58.997942 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:33:58.997914    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5f35fd_4043_4391_98ac_2404de3c59b6.slice/crio-e3432533367d631bdefe4435a73600f4bdb63f0f47997634ad903da34c1eaeba WatchSource:0}: Error finding container e3432533367d631bdefe4435a73600f4bdb63f0f47997634ad903da34c1eaeba: Status 404 returned error can't find the container with id e3432533367d631bdefe4435a73600f4bdb63f0f47997634ad903da34c1eaeba
Nov 03 01:33:59.130647 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.130617    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"]
Nov 03 01:33:59.131839 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:33:59.131813    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a688b9c_7e43_4c79_bfb0_c626dbdbf408.slice/crio-b420ed00ce2f4727565acb8ce8fcd6cca2b54c11633492e3a138c8b4d7da8fc1 WatchSource:0}: Error finding container b420ed00ce2f4727565acb8ce8fcd6cca2b54c11633492e3a138c8b4d7da8fc1: Status 404 returned error can't find the container with id b420ed00ce2f4727565acb8ce8fcd6cca2b54c11633492e3a138c8b4d7da8fc1
Nov 03 01:33:59.933434 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.933385    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np" event={"ID":"2a02b1db-df08-4339-a68a-6b92094afde3","Type":"ContainerStarted","Data":"697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd"}
Nov 03 01:33:59.933904 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.933506    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:33:59.935979 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.935945    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42" event={"ID":"4a688b9c-7e43-4c79-bfb0-c626dbdbf408","Type":"ContainerStarted","Data":"a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1"}
Nov 03 01:33:59.936150 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.936008    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42" event={"ID":"4a688b9c-7e43-4c79-bfb0-c626dbdbf408","Type":"ContainerStarted","Data":"b420ed00ce2f4727565acb8ce8fcd6cca2b54c11633492e3a138c8b4d7da8fc1"}
Nov 03 01:33:59.936150 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.936044    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:33:59.937826 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.937803    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd" event={"ID":"7e5f35fd-4043-4391-98ac-2404de3c59b6","Type":"ContainerStarted","Data":"78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6"}
Nov 03 01:33:59.937929 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.937840    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd" event={"ID":"7e5f35fd-4043-4391-98ac-2404de3c59b6","Type":"ContainerStarted","Data":"e3432533367d631bdefe4435a73600f4bdb63f0f47997634ad903da34c1eaeba"}
Nov 03 01:33:59.937929 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.937871    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:33:59.971161 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:33:59.971103    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np" podStartSLOduration=2.574863929 podStartE2EDuration="2.971089149s" totalImagesPullingTime="396.22522ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:33:59.969550148 +0000 UTC m=+5277.524902186" watchObservedRunningTime="2026-11-03 01:33:59.971089149 +0000 UTC m=+5277.526441173"
Nov 03 01:34:00.005310 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:00.005235    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd" podStartSLOduration=2.005222237 podStartE2EDuration="2.005222237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:00.004628272 +0000 UTC m=+5277.559980297" watchObservedRunningTime="2026-11-03 01:34:00.005222237 +0000 UTC m=+5277.560574262"
Nov 03 01:34:00.041037 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:00.040964    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42" podStartSLOduration=2.040946598 podStartE2EDuration="2.040946598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:33:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:34:00.039549648 +0000 UTC m=+5277.594901671" watchObservedRunningTime="2026-11-03 01:34:00.040946598 +0000 UTC m=+5277.596298623"
Nov 03 01:34:03.806412 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:03.806370    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:34:05.948161 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:05.948133    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:34:10.945577 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:10.945547    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:34:10.946102 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:10.945625    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:34:17.036863 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:17.036825    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"]
Nov 03 01:34:17.066643 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:17.066609    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"]
Nov 03 01:34:18.506143 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.506103    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"]
Nov 03 01:34:18.506805 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.506356    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np" podUID="2a02b1db-df08-4339-a68a-6b92094afde3" containerName="maas-api" containerID="cri-o://697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd" gracePeriod=30
Nov 03 01:34:18.782226 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.782145    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:34:18.837086 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.837056    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a02b1db-df08-4339-a68a-6b92094afde3-kube-api-access-r2gcg\" (UniqueName: \"kubernetes.io/projected/2a02b1db-df08-4339-a68a-6b92094afde3-kube-api-access-r2gcg\") pod \"2a02b1db-df08-4339-a68a-6b92094afde3\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") "
Nov 03 01:34:18.837313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.837119    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-metrics-certs\") pod \"2a02b1db-df08-4339-a68a-6b92094afde3\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") "
Nov 03 01:34:18.837313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.837161    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a02b1db-df08-4339-a68a-6b92094afde3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a02b1db-df08-4339-a68a-6b92094afde3-openshift-service-ca\") pod \"2a02b1db-df08-4339-a68a-6b92094afde3\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") "
Nov 03 01:34:18.837313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.837210    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-maas-api-tls\") pod \"2a02b1db-df08-4339-a68a-6b92094afde3\" (UID: \"2a02b1db-df08-4339-a68a-6b92094afde3\") "
Nov 03 01:34:18.837830 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.837797    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a02b1db-df08-4339-a68a-6b92094afde3-openshift-service-ca" pod "2a02b1db-df08-4339-a68a-6b92094afde3" (UID: "2a02b1db-df08-4339-a68a-6b92094afde3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:18.840006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.839982    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-metrics-certs" pod "2a02b1db-df08-4339-a68a-6b92094afde3" (UID: "2a02b1db-df08-4339-a68a-6b92094afde3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:18.840316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.840293    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-maas-api-tls" pod "2a02b1db-df08-4339-a68a-6b92094afde3" (UID: "2a02b1db-df08-4339-a68a-6b92094afde3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:34:18.840416 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.840322    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a02b1db-df08-4339-a68a-6b92094afde3-kube-api-access-r2gcg" pod "2a02b1db-df08-4339-a68a-6b92094afde3" (UID: "2a02b1db-df08-4339-a68a-6b92094afde3"). InnerVolumeSpecName "kube-api-access-r2gcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:18.938599 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.938557    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2gcg\" (UniqueName: \"kubernetes.io/projected/2a02b1db-df08-4339-a68a-6b92094afde3-kube-api-access-r2gcg\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:18.938890 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.938862    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:18.939008 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.938995    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a02b1db-df08-4339-a68a-6b92094afde3-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:18.939096 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:18.939085    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2a02b1db-df08-4339-a68a-6b92094afde3-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:19.033632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.033542    6000 generic.go:356] "Generic (PLEG): container finished" podID="2a02b1db-df08-4339-a68a-6b92094afde3" containerID="697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd" exitCode=0
Nov 03 01:34:19.033632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.033583    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np" event={"ID":"2a02b1db-df08-4339-a68a-6b92094afde3","Type":"ContainerDied","Data":"697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd"}
Nov 03 01:34:19.033632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.033594    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"
Nov 03 01:34:19.033632 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.033624    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np" event={"ID":"2a02b1db-df08-4339-a68a-6b92094afde3","Type":"ContainerDied","Data":"88d53e143912f08648b448942bab2c8afbe740165a72cda6cdf52b53a9cca493"}
Nov 03 01:34:19.033979 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.033644    6000 scope.go:122] "RemoveContainer" containerID="697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd"
Nov 03 01:34:19.043536 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.043516    6000 scope.go:122] "RemoveContainer" containerID="697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd"
Nov 03 01:34:19.043821 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:34:19.043802    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd\": container with ID starting with 697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd not found: ID does not exist" containerID="697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd"
Nov 03 01:34:19.043875 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.043831    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd"} err="failed to get container status \"697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd\": rpc error: code = NotFound desc = could not find container \"697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd\": container with ID starting with 697d25a400e1a90f250f2c536e9017d7ed270eb068d33e08cd417514b64dcfcd not found: ID does not exist"
Nov 03 01:34:19.080917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.080885    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"]
Nov 03 01:34:19.092597 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.092568    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-462c10-85c9f65d44-fj9np"]
Nov 03 01:34:19.628518 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.628477    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"]
Nov 03 01:34:19.628968 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:19.628739    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy" containerID="cri-o://0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a" gracePeriod=30
Nov 03 01:34:19.896774 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:34:19.896693    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:34:20.566142 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:20.566108    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a02b1db-df08-4339-a68a-6b92094afde3" path="/var/lib/kubelet/pods/2a02b1db-df08-4339-a68a-6b92094afde3/volumes"
Nov 03 01:34:21.340280 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:21.340224    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:34:22.037649 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.037583    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42" podUID="4a688b9c-7e43-4c79-bfb0-c626dbdbf408" containerName="payload-processing" containerID="cri-o://a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1" gracePeriod=25
Nov 03 01:34:22.067200 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.067165    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd" podUID="7e5f35fd-4043-4391-98ac-2404de3c59b6" containerName="payload-pre-processing" containerID="cri-o://78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6" gracePeriod=25
Nov 03 01:34:22.321818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.321789    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:34:22.351098 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.351066    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:34:22.475979 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.475948    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-kube-api-access-mwbxp\" (UniqueName: \"kubernetes.io/projected/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-kube-api-access-mwbxp\") pod \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") "
Nov 03 01:34:22.476194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.475998    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e5f35fd-4043-4391-98ac-2404de3c59b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5f35fd-4043-4391-98ac-2404de3c59b6-plugins-config-volume\") pod \"7e5f35fd-4043-4391-98ac-2404de3c59b6\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") "
Nov 03 01:34:22.476194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.476019    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-plugins-config-volume\") pod \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\" (UID: \"4a688b9c-7e43-4c79-bfb0-c626dbdbf408\") "
Nov 03 01:34:22.476194 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.476074    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e5f35fd-4043-4391-98ac-2404de3c59b6-kube-api-access-7b228\" (UniqueName: \"kubernetes.io/projected/7e5f35fd-4043-4391-98ac-2404de3c59b6-kube-api-access-7b228\") pod \"7e5f35fd-4043-4391-98ac-2404de3c59b6\" (UID: \"7e5f35fd-4043-4391-98ac-2404de3c59b6\") "
Nov 03 01:34:22.476454 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.476427    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-plugins-config-volume" pod "4a688b9c-7e43-4c79-bfb0-c626dbdbf408" (UID: "4a688b9c-7e43-4c79-bfb0-c626dbdbf408"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:22.476513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.476476    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5f35fd-4043-4391-98ac-2404de3c59b6-plugins-config-volume" pod "7e5f35fd-4043-4391-98ac-2404de3c59b6" (UID: "7e5f35fd-4043-4391-98ac-2404de3c59b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:22.476561 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.476525    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:22.478139 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.478123    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-kube-api-access-mwbxp" pod "4a688b9c-7e43-4c79-bfb0-c626dbdbf408" (UID: "4a688b9c-7e43-4c79-bfb0-c626dbdbf408"). InnerVolumeSpecName "kube-api-access-mwbxp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:22.478420 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.478402    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e5f35fd-4043-4391-98ac-2404de3c59b6-kube-api-access-7b228" pod "7e5f35fd-4043-4391-98ac-2404de3c59b6" (UID: "7e5f35fd-4043-4391-98ac-2404de3c59b6"). InnerVolumeSpecName "kube-api-access-7b228". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:22.578731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.578571    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwbxp\" (UniqueName: \"kubernetes.io/projected/4a688b9c-7e43-4c79-bfb0-c626dbdbf408-kube-api-access-mwbxp\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:22.578731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.578606    6000 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e5f35fd-4043-4391-98ac-2404de3c59b6-plugins-config-volume\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:22.578731 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:22.578625    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7b228\" (UniqueName: \"kubernetes.io/projected/7e5f35fd-4043-4391-98ac-2404de3c59b6-kube-api-access-7b228\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:23.056402 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.056369    6000 generic.go:356] "Generic (PLEG): container finished" podID="4a688b9c-7e43-4c79-bfb0-c626dbdbf408" containerID="a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1" exitCode=0
Nov 03 01:34:23.056570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.056438    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"
Nov 03 01:34:23.056570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.056455    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42" event={"ID":"4a688b9c-7e43-4c79-bfb0-c626dbdbf408","Type":"ContainerDied","Data":"a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1"}
Nov 03 01:34:23.056570 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.056545    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42" event={"ID":"4a688b9c-7e43-4c79-bfb0-c626dbdbf408","Type":"ContainerDied","Data":"b420ed00ce2f4727565acb8ce8fcd6cca2b54c11633492e3a138c8b4d7da8fc1"}
Nov 03 01:34:23.056728 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.056571    6000 scope.go:122] "RemoveContainer" containerID="a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1"
Nov 03 01:34:23.058176 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.058151    6000 generic.go:356] "Generic (PLEG): container finished" podID="7e5f35fd-4043-4391-98ac-2404de3c59b6" containerID="78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6" exitCode=0
Nov 03 01:34:23.058323 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.058240    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"
Nov 03 01:34:23.058323 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.058305    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd" event={"ID":"7e5f35fd-4043-4391-98ac-2404de3c59b6","Type":"ContainerDied","Data":"78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6"}
Nov 03 01:34:23.058474 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.058343    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd" event={"ID":"7e5f35fd-4043-4391-98ac-2404de3c59b6","Type":"ContainerDied","Data":"e3432533367d631bdefe4435a73600f4bdb63f0f47997634ad903da34c1eaeba"}
Nov 03 01:34:23.065651 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.065634    6000 scope.go:122] "RemoveContainer" containerID="a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1"
Nov 03 01:34:23.065926 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:34:23.065907    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1\": container with ID starting with a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1 not found: ID does not exist" containerID="a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1"
Nov 03 01:34:23.065976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.065933    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1"} err="failed to get container status \"a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1\": rpc error: code = NotFound desc = could not find container \"a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1\": container with ID starting with a4e8949c18c61e01e0060f531f7b9ebb61049e93e0c22ae5153fa815c3dca0c1 not found: ID does not exist"
Nov 03 01:34:23.065976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.065947    6000 scope.go:122] "RemoveContainer" containerID="78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6"
Nov 03 01:34:23.074069 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.074046    6000 scope.go:122] "RemoveContainer" containerID="78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6"
Nov 03 01:34:23.074353 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:34:23.074328    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6\": container with ID starting with 78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6 not found: ID does not exist" containerID="78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6"
Nov 03 01:34:23.074421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.074362    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6"} err="failed to get container status \"78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6\": rpc error: code = NotFound desc = could not find container \"78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6\": container with ID starting with 78adb78269ec32ecb95772d483f7499687b3e5285aeb688cd484fd2f85ec8ee6 not found: ID does not exist"
Nov 03 01:34:23.104313 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.104279    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"]
Nov 03 01:34:23.115761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.115727    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-462c10-6448dpt8nd"]
Nov 03 01:34:23.138273 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.138230    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"]
Nov 03 01:34:23.140095 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:23.140066    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-462c10-6b787884fbwd42"]
Nov 03 01:34:24.565917 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:24.565838    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a688b9c-7e43-4c79-bfb0-c626dbdbf408" path="/var/lib/kubelet/pods/4a688b9c-7e43-4c79-bfb0-c626dbdbf408/volumes"
Nov 03 01:34:24.566423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:24.566320    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e5f35fd-4043-4391-98ac-2404de3c59b6" path="/var/lib/kubelet/pods/7e5f35fd-4043-4391-98ac-2404de3c59b6/volumes"
Nov 03 01:34:24.882996 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:24.882970    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:34:25.004575 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004536    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-credential-socket\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.004789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004593    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-socket\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.004789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004622    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32a833f1-f2e7-488e-b974-28e63fd248f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a833f1-f2e7-488e-b974-28e63fd248f1-istiod-ca-cert\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.004789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004668    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-envoy\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.004789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004745    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-certs\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.004789 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004790    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-data\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.005119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004848    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-podinfo\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.005119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004909    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-credential-socket" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:25.005119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.004921    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-token\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-token\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.005119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005008    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-socket" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:25.005119 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005044    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-kube-api-access-ljmvk\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-kube-api-access-ljmvk\") pod \"32a833f1-f2e7-488e-b974-28e63fd248f1\" (UID: \"32a833f1-f2e7-488e-b974-28e63fd248f1\") "
Nov 03 01:34:25.005437 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005128    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-certs" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:25.005437 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005152    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32a833f1-f2e7-488e-b974-28e63fd248f1-istiod-ca-cert" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:34:25.005437 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005411    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-data" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:25.005590 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005471    6000 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-credential-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.005590 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005490    6000 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-socket\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.005590 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005508    6000 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32a833f1-f2e7-488e-b974-28e63fd248f1-istiod-ca-cert\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.005590 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.005526    6000 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-workload-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.007468 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.007440    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-kube-api-access-ljmvk" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "kube-api-access-ljmvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:25.007603 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.007483    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-token" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:34:25.007721 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.007663    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-envoy" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 03 01:34:25.007721 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.007668    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-podinfo" pod "32a833f1-f2e7-488e-b974-28e63fd248f1" (UID: "32a833f1-f2e7-488e-b974-28e63fd248f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 03 01:34:25.074215 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.074182    6000 generic.go:356] "Generic (PLEG): container finished" podID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerID="0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a" exitCode=0
Nov 03 01:34:25.074423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.074231    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"
Nov 03 01:34:25.074423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.074281    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" event={"ID":"32a833f1-f2e7-488e-b974-28e63fd248f1","Type":"ContainerDied","Data":"0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a"}
Nov 03 01:34:25.074423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.074347    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw" event={"ID":"32a833f1-f2e7-488e-b974-28e63fd248f1","Type":"ContainerDied","Data":"e0b2086336040df8fbda7f4367a032f7ab99e6c0cff350a84c0e78a168deb7c4"}
Nov 03 01:34:25.074423 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.074380    6000 scope.go:122] "RemoveContainer" containerID="0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a"
Nov 03 01:34:25.083627 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.083610    6000 scope.go:122] "RemoveContainer" containerID="923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b"
Nov 03 01:34:25.091976 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.091958    6000 scope.go:122] "RemoveContainer" containerID="0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a"
Nov 03 01:34:25.092235 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:34:25.092217    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a\": container with ID starting with 0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a not found: ID does not exist" containerID="0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a"
Nov 03 01:34:25.092421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.092244    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a"} err="failed to get container status \"0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a\": rpc error: code = NotFound desc = could not find container \"0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a\": container with ID starting with 0184a196cc91e18f4d5557d6bd389c111cab63e2e4fd94d2104ae7b79fa4704a not found: ID does not exist"
Nov 03 01:34:25.092421 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.092280    6000 scope.go:122] "RemoveContainer" containerID="923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b"
Nov 03 01:34:25.092520 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:34:25.092480    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b\": container with ID starting with 923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b not found: ID does not exist" containerID="923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b"
Nov 03 01:34:25.092520 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.092501    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b"} err="failed to get container status \"923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b\": rpc error: code = NotFound desc = could not find container \"923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b\": container with ID starting with 923a186620ecde76655c912d3f91132b71c822f989ed1c730fff64c2fd99012b not found: ID does not exist"
Nov 03 01:34:25.106064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.106044    6000 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-envoy\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.106064 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.106065    6000 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-data\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.106190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.106074    6000 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-podinfo\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.106190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.106081    6000 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-istio-token\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.106190 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.106089    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljmvk\" (UniqueName: \"kubernetes.io/projected/32a833f1-f2e7-488e-b974-28e63fd248f1-kube-api-access-ljmvk\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:34:25.121822 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.121756    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"]
Nov 03 01:34:25.127470 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:25.127446    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-462c10-openshift-default-67dc98df7c8qhlw"]
Nov 03 01:34:26.566012 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:26.565964    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" path="/var/lib/kubelet/pods/32a833f1-f2e7-488e-b974-28e63fd248f1/volumes"
Nov 03 01:34:33.039734 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:33.039697    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:34:38.510006 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:34:38.509971    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:34:49.895456 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:34:49.895414    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:35:19.894810 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:35:19.894761    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:35:49.895382 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:35:49.895345    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:36:04.882221 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:04.882166    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:36:17.607502 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:17.607410    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:36:19.895845 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:36:19.895809    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:36:22.760032 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:22.759989    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:36:27.129181 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:27.129146    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:36:33.492391 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492352    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"]
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492792    6000 cpu_manager.go:414] "containerMap: removing container" podUID="7e5f35fd-4043-4391-98ac-2404de3c59b6" containerName="payload-pre-processing"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492805    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e5f35fd-4043-4391-98ac-2404de3c59b6" containerName="payload-pre-processing"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492821    6000 cpu_manager.go:414] "containerMap: removing container" podUID="2a02b1db-df08-4339-a68a-6b92094afde3" containerName="maas-api"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492826    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a02b1db-df08-4339-a68a-6b92094afde3" containerName="maas-api"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492835    6000 cpu_manager.go:414] "containerMap: removing container" podUID="4a688b9c-7e43-4c79-bfb0-c626dbdbf408" containerName="payload-processing"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492840    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a688b9c-7e43-4c79-bfb0-c626dbdbf408" containerName="payload-processing"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492855    6000 cpu_manager.go:414] "containerMap: removing container" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492861    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492874    6000 cpu_manager.go:414] "containerMap: removing container" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy"
Nov 03 01:36:33.492921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492879    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy"
Nov 03 01:36:33.493331 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492945    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy"
Nov 03 01:36:33.493331 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492957    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e5f35fd-4043-4391-98ac-2404de3c59b6" containerName="payload-pre-processing"
Nov 03 01:36:33.493331 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492963    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a02b1db-df08-4339-a68a-6b92094afde3" containerName="maas-api"
Nov 03 01:36:33.493331 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.492969    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a688b9c-7e43-4c79-bfb0-c626dbdbf408" containerName="payload-processing"
Nov 03 01:36:33.496448 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.496425    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.525583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.525537    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"]
Nov 03 01:36:33.606198 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.606158    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7095c2d6-e48f-457f-9221-44fa5af9543b-openshift-service-ca\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.606399 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.606357    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7095c2d6-e48f-457f-9221-44fa5af9543b-metrics-certs\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.606454 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.606434    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7095c2d6-e48f-457f-9221-44fa5af9543b-maas-api-tls\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.606586 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.606560    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-964fk\" (UniqueName: \"kubernetes.io/projected/7095c2d6-e48f-457f-9221-44fa5af9543b-kube-api-access-964fk\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.707744 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.707710    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7095c2d6-e48f-457f-9221-44fa5af9543b-openshift-service-ca\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.707744 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.707747    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7095c2d6-e48f-457f-9221-44fa5af9543b-metrics-certs\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.707982 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.707808    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7095c2d6-e48f-457f-9221-44fa5af9543b-maas-api-tls\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.707982 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.707833    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-964fk\" (UniqueName: \"kubernetes.io/projected/7095c2d6-e48f-457f-9221-44fa5af9543b-kube-api-access-964fk\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.708452 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.708414    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7095c2d6-e48f-457f-9221-44fa5af9543b-openshift-service-ca\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.710399 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.710377    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7095c2d6-e48f-457f-9221-44fa5af9543b-metrics-certs\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.710574 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.710552    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7095c2d6-e48f-457f-9221-44fa5af9543b-maas-api-tls\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.725564 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.725541    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-964fk\" (UniqueName: \"kubernetes.io/projected/7095c2d6-e48f-457f-9221-44fa5af9543b-kube-api-access-964fk\") pod \"maas-api-6cdc687864-gqv6x\" (UID: \"7095c2d6-e48f-457f-9221-44fa5af9543b\") " pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:33.816601 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:33.816510    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:34.109637 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:34.109607    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"]
Nov 03 01:36:34.111226 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:36:34.111191    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7095c2d6_e48f_457f_9221_44fa5af9543b.slice/crio-a12978138408415cea19b996f64f8f159c99bc56c2997329fe7c280e80669b8d WatchSource:0}: Error finding container a12978138408415cea19b996f64f8f159c99bc56c2997329fe7c280e80669b8d: Status 404 returned error can't find the container with id a12978138408415cea19b996f64f8f159c99bc56c2997329fe7c280e80669b8d
Nov 03 01:36:34.773743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:34.773657    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x" event={"ID":"7095c2d6-e48f-457f-9221-44fa5af9543b","Type":"ContainerStarted","Data":"681ec30d76bab16ef3bfb6921ace196d6a0c2ea59cf76e4a9cf8892d85d778d5"}
Nov 03 01:36:34.773743 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:34.773721    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:34.774158 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:34.773746    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x" event={"ID":"7095c2d6-e48f-457f-9221-44fa5af9543b","Type":"ContainerStarted","Data":"a12978138408415cea19b996f64f8f159c99bc56c2997329fe7c280e80669b8d"}
Nov 03 01:36:34.812004 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:34.811939    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x" podStartSLOduration=1.450137893 podStartE2EDuration="1.811924442s" totalImagesPullingTime="361.786549ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:36:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:36:34.810643111 +0000 UTC m=+5432.365995136" watchObservedRunningTime="2026-11-03 01:36:34.811924442 +0000 UTC m=+5432.367276480"
Nov 03 01:36:40.784325 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:40.784298    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cdc687864-gqv6x"
Nov 03 01:36:44.811034 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:44.810994    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:36:48.295448 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:48.295415    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-7gw56"]
Nov 03 01:36:48.295959 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:48.295653    6000 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56" podUID="d3701479-3c25-4485-8630-2e25c3950943" containerName="maas-api" containerID="cri-o://e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b" gracePeriod=30
Nov 03 01:36:49.149950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.149900    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56" podUID="d3701479-3c25-4485-8630-2e25c3950943" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.31:8443/health\": dial tcp 10.134.0.31:8443: connect: connection refused"
Nov 03 01:36:49.609334 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.608969    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 01:36:49.711409 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.711307    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-metrics-certs\") pod \"d3701479-3c25-4485-8630-2e25c3950943\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") "
Nov 03 01:36:49.711767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.711744    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-maas-api-tls\") pod \"d3701479-3c25-4485-8630-2e25c3950943\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") "
Nov 03 01:36:49.712312 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.712284    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3701479-3c25-4485-8630-2e25c3950943-kube-api-access-nqb58\" (UniqueName: \"kubernetes.io/projected/d3701479-3c25-4485-8630-2e25c3950943-kube-api-access-nqb58\") pod \"d3701479-3c25-4485-8630-2e25c3950943\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") "
Nov 03 01:36:49.712668 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.712539    6000 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3701479-3c25-4485-8630-2e25c3950943-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3701479-3c25-4485-8630-2e25c3950943-openshift-service-ca\") pod \"d3701479-3c25-4485-8630-2e25c3950943\" (UID: \"d3701479-3c25-4485-8630-2e25c3950943\") "
Nov 03 01:36:49.713070 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.713032    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3701479-3c25-4485-8630-2e25c3950943-openshift-service-ca" pod "d3701479-3c25-4485-8630-2e25c3950943" (UID: "d3701479-3c25-4485-8630-2e25c3950943"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 03 01:36:49.713481 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.713458    6000 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3701479-3c25-4485-8630-2e25c3950943-openshift-service-ca\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:36:49.717991 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.717944    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-metrics-certs" pod "d3701479-3c25-4485-8630-2e25c3950943" (UID: "d3701479-3c25-4485-8630-2e25c3950943"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:49.718115 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.717978    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-maas-api-tls" pod "d3701479-3c25-4485-8630-2e25c3950943" (UID: "d3701479-3c25-4485-8630-2e25c3950943"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 03 01:36:49.719752 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.719722    6000 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3701479-3c25-4485-8630-2e25c3950943-kube-api-access-nqb58" pod "d3701479-3c25-4485-8630-2e25c3950943" (UID: "d3701479-3c25-4485-8630-2e25c3950943"). InnerVolumeSpecName "kube-api-access-nqb58". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 03 01:36:49.815081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.815040    6000 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-maas-api-tls\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:36:49.815081 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.815083    6000 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqb58\" (UniqueName: \"kubernetes.io/projected/d3701479-3c25-4485-8630-2e25c3950943-kube-api-access-nqb58\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:36:49.815414 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.815102    6000 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3701479-3c25-4485-8630-2e25c3950943-metrics-certs\") on node \"ip-10-0-136-97.ec2.internal\" DevicePath \"\""
Nov 03 01:36:49.875282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.874771    6000 generic.go:356] "Generic (PLEG): container finished" podID="d3701479-3c25-4485-8630-2e25c3950943" containerID="e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b" exitCode=0
Nov 03 01:36:49.875282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.874856    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56" event={"ID":"d3701479-3c25-4485-8630-2e25c3950943","Type":"ContainerDied","Data":"e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b"}
Nov 03 01:36:49.875282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.874916    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56" event={"ID":"d3701479-3c25-4485-8630-2e25c3950943","Type":"ContainerDied","Data":"1ef2b285a554d26845ecb1356ab1235a5d02ce946d5a770e9de30d703cb6e961"}
Nov 03 01:36:49.875282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.874942    6000 scope.go:122] "RemoveContainer" containerID="e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b"
Nov 03 01:36:49.875282 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.875092    6000 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6675486f96-7gw56"
Nov 03 01:36:49.947484 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.936110    6000 scope.go:122] "RemoveContainer" containerID="e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b"
Nov 03 01:36:49.947484 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:36:49.944783    6000 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b\": container with ID starting with e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b not found: ID does not exist" containerID="e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b"
Nov 03 01:36:49.947484 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.944827    6000 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b"} err="failed to get container status \"e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b\": rpc error: code = NotFound desc = could not find container \"e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b\": container with ID starting with e72aa12c2c5958e54d3c06eeecfb27324ae2e05c714e96b62806b8226aed0b3b not found: ID does not exist"
Nov 03 01:36:49.947484 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.945387    6000 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-7gw56"]
Nov 03 01:36:49.985340 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:49.980284    6000 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6675486f96-7gw56"]
Nov 03 01:36:50.016281 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:36:50.015468    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:36:50.583715 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:36:50.583679    6000 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3701479-3c25-4485-8630-2e25c3950943" path="/var/lib/kubelet/pods/d3701479-3c25-4485-8630-2e25c3950943/volumes"
Nov 03 01:37:14.696719 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:37:14.696680    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:37:19.894839 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:37:19.894792    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:37:30.404708 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:37:30.404669    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:37:47.608646 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:37:47.608562    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:37:49.895673 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:37:49.895636    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:37:52.199242 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:37:52.199199    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:37:58.732358 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:37:58.732325    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:38:18.424593 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:38:18.424558    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:38:19.895320 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:38:19.895284    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:38:28.712013 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:38:28.711982    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:38:33.224933 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:38:33.224889    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:38:41.617583 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:38:41.617549    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:38:49.894646 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:38:49.894608    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:39:00.150624 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:39:00.150585    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:39:10.628819 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:39:10.628785    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:39:15.845127 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:39:15.844920    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:39:19.896089 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:39:19.896048    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:39:26.031761 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:39:26.031728    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:39:43.622616 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:39:43.622581    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:39:49.894526 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:39:49.894487    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:40:19.894679 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:40:19.894640    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:40:38.618281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:40:38.617367    6000 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-6cc458d887-msgxv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 03 01:40:38.618281 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:40:38.617443    6000 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-6cc458d887-msgxv" podUID="f65d6c3e-af31-47a7-a807-15531faca35f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 03 01:40:49.894486 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:40:49.894404    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:41:20.022789 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:41:20.022710    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:41:37.719275 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:41:37.719220    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-kxq2l"]
Nov 03 01:41:49.894823 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:41:49.894784    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:42:00.913109 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:00.913076    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77bbb667f4-ps8vw_ec8b5850-10d2-4e94-96fc-834119b90561/manager/0.log"
Nov 03 01:42:01.841970 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:01.841934    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc8c7d7fb-gcc5b_4146cae1-50a3-4d36-b7af-65bb9c50655d/manager/0.log"
Nov 03 01:42:02.520302 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:02.520249    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hm874_6f731881-a91d-48e5-b881-8bee6561c7fb/proxy/0.log"
Nov 03 01:42:03.426427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:03.426393    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6cdc687864-gqv6x_7095c2d6-e48f-457f-9221-44fa5af9543b/maas-api/0.log"
Nov 03 01:42:10.132345 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:10.132311    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kxq2l_f109704e-d9c1-41e3-84c9-5a0aa3eac6be/limitador/0.log"
Nov 03 01:42:12.059698 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:12.059619    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-hlqrl_7be7a5b4-d36a-48a6-afc8-40741783c4fa/kube-auth-proxy/0.log"
Nov 03 01:42:12.681143 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:12.681110    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-dhv97_ad946480-1aaa-4a6c-826c-a0db0c4a9ac9/istio-proxy/0.log"
Nov 03 01:42:13.456985 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:13.456951    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-msgxv_f65d6c3e-af31-47a7-a807-15531faca35f/istio-proxy/0.log"
Nov 03 01:42:13.867269 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:13.867227    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-t297d_ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf/istio-proxy/0.log"
Nov 03 01:42:14.065302 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:14.065249    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-w99jj_b869da23-2c4e-4e69-a1b7-a9ae8585d8a1/istio-proxy/0.log"
Nov 03 01:42:14.458703 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:14.458671    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-75b7db44c9-qzv2n_0c0d40bf-fa16-4c0e-aa3c-595693ee1513/payload-pre-processing/0.log"
Nov 03 01:42:14.662183 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:14.662153    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-846db64bd8-6tnsj_9949daa4-550e-4e71-bdb2-7d3f046eeb9d/payload-processing/0.log"
Nov 03 01:42:15.800477 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:15.800445    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mnz29_04cc1bb4-bbe7-468b-9e49-8df72c661ee1/main/0.log"
Nov 03 01:42:16.197337 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:16.197306    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-m2dbv_f6adde23-6bf2-4a92-8002-a6e093c1c2ba/main/0.log"
Nov 03 01:42:16.761046 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:16.761011    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-lfqwv_f61b18a7-80a7-431a-9a91-ac141ff2bcdf/main/0.log"
Nov 03 01:42:16.946883 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:16.946832    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-fj4kk_772fdde9-c1c8-47a0-ada4-d34e8033de8b/main/0.log"
Nov 03 01:42:19.895566 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:42:19.895528    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:42:27.829505 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.829460    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n428r/must-gather-95vgz"]
Nov 03 01:42:27.830105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.829909    6000 cpu_manager.go:414] "containerMap: removing container" podUID="d3701479-3c25-4485-8630-2e25c3950943" containerName="maas-api"
Nov 03 01:42:27.830105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.829920    6000 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3701479-3c25-4485-8630-2e25c3950943" containerName="maas-api"
Nov 03 01:42:27.830105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.830008    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="32a833f1-f2e7-488e-b974-28e63fd248f1" containerName="istio-proxy"
Nov 03 01:42:27.830105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.830020    6000 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3701479-3c25-4485-8630-2e25c3950943" containerName="maas-api"
Nov 03 01:42:27.833443 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.833422    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-68v8q/must-gather-xvpbg"]
Nov 03 01:42:27.833592 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.833562    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:27.837379 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.837359    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:27.848017 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.848000    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-n428r\"/\"default-dockercfg-dmzvq\""
Nov 03 01:42:27.848307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.848289    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n428r\"/\"kube-root-ca.crt\""
Nov 03 01:42:27.848307 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.848294    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-n428r\"/\"openshift-service-ca.crt\""
Nov 03 01:42:27.859831 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.859803    6000 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-68v8q\"/\"default-dockercfg-dkjcg\""
Nov 03 01:42:27.861408 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.861387    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/must-gather-95vgz"]
Nov 03 01:42:27.877145 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.877100    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68v8q/must-gather-xvpbg"]
Nov 03 01:42:27.892476 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.892451    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68v8q\"/\"openshift-service-ca.crt\""
Nov 03 01:42:27.892611 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.892542    6000 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-68v8q\"/\"kube-root-ca.crt\""
Nov 03 01:42:27.900189 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.900167    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s95wp\" (UniqueName: \"kubernetes.io/projected/7a457d0a-4dcf-48d7-8e33-f528443f95f9-kube-api-access-s95wp\") pod \"must-gather-xvpbg\" (UID: \"7a457d0a-4dcf-48d7-8e33-f528443f95f9\") " pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:27.900319 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.900214    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d162a947-29f0-4f90-a650-fd109e6633bc-must-gather-output\") pod \"must-gather-95vgz\" (UID: \"d162a947-29f0-4f90-a650-fd109e6633bc\") " pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:27.900319 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.900307    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a457d0a-4dcf-48d7-8e33-f528443f95f9-must-gather-output\") pod \"must-gather-xvpbg\" (UID: \"7a457d0a-4dcf-48d7-8e33-f528443f95f9\") " pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:27.900419 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:27.900365    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkxlf\" (UniqueName: \"kubernetes.io/projected/d162a947-29f0-4f90-a650-fd109e6633bc-kube-api-access-mkxlf\") pod \"must-gather-95vgz\" (UID: \"d162a947-29f0-4f90-a650-fd109e6633bc\") " pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:28.001679 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.001644    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s95wp\" (UniqueName: \"kubernetes.io/projected/7a457d0a-4dcf-48d7-8e33-f528443f95f9-kube-api-access-s95wp\") pod \"must-gather-xvpbg\" (UID: \"7a457d0a-4dcf-48d7-8e33-f528443f95f9\") " pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:28.001903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.001696    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d162a947-29f0-4f90-a650-fd109e6633bc-must-gather-output\") pod \"must-gather-95vgz\" (UID: \"d162a947-29f0-4f90-a650-fd109e6633bc\") " pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:28.001903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.001723    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a457d0a-4dcf-48d7-8e33-f528443f95f9-must-gather-output\") pod \"must-gather-xvpbg\" (UID: \"7a457d0a-4dcf-48d7-8e33-f528443f95f9\") " pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:28.001903 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.001760    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkxlf\" (UniqueName: \"kubernetes.io/projected/d162a947-29f0-4f90-a650-fd109e6633bc-kube-api-access-mkxlf\") pod \"must-gather-95vgz\" (UID: \"d162a947-29f0-4f90-a650-fd109e6633bc\") " pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:28.002103 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.002078    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/d162a947-29f0-4f90-a650-fd109e6633bc-must-gather-output\") pod \"must-gather-95vgz\" (UID: \"d162a947-29f0-4f90-a650-fd109e6633bc\") " pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:28.002162 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.002088    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7a457d0a-4dcf-48d7-8e33-f528443f95f9-must-gather-output\") pod \"must-gather-xvpbg\" (UID: \"7a457d0a-4dcf-48d7-8e33-f528443f95f9\") " pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:28.021896 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.021859    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s95wp\" (UniqueName: \"kubernetes.io/projected/7a457d0a-4dcf-48d7-8e33-f528443f95f9-kube-api-access-s95wp\") pod \"must-gather-xvpbg\" (UID: \"7a457d0a-4dcf-48d7-8e33-f528443f95f9\") " pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:28.022049 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.022026    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkxlf\" (UniqueName: \"kubernetes.io/projected/d162a947-29f0-4f90-a650-fd109e6633bc-kube-api-access-mkxlf\") pod \"must-gather-95vgz\" (UID: \"d162a947-29f0-4f90-a650-fd109e6633bc\") " pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:28.150553 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.150509    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/must-gather-95vgz"
Nov 03 01:42:28.155696 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.155677    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-68v8q/must-gather-xvpbg"
Nov 03 01:42:28.451302 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.451275    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-68v8q/must-gather-xvpbg"]
Nov 03 01:42:28.452807 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:42:28.452780    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7a457d0a_4dcf_48d7_8e33_f528443f95f9.slice/crio-efdc3d0f26b1fae0f5c2433a708892a938ee353247b78a0cabe51badbd6bd41f WatchSource:0}: Error finding container efdc3d0f26b1fae0f5c2433a708892a938ee353247b78a0cabe51badbd6bd41f: Status 404 returned error can't find the container with id efdc3d0f26b1fae0f5c2433a708892a938ee353247b78a0cabe51badbd6bd41f
Nov 03 01:42:28.454811 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.454787    6000 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 03 01:42:28.464779 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:28.461325    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/must-gather-95vgz"]
Nov 03 01:42:28.464779 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:42:28.464447    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd162a947_29f0_4f90_a650_fd109e6633bc.slice/crio-2225f4a645584a2c4f8ea303993c5e6476659f87235068d46e7ae6a90a370c22 WatchSource:0}: Error finding container 2225f4a645584a2c4f8ea303993c5e6476659f87235068d46e7ae6a90a370c22: Status 404 returned error can't find the container with id 2225f4a645584a2c4f8ea303993c5e6476659f87235068d46e7ae6a90a370c22
Nov 03 01:42:29.267947 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:29.267908    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/must-gather-95vgz" event={"ID":"d162a947-29f0-4f90-a650-fd109e6633bc","Type":"ContainerStarted","Data":"2225f4a645584a2c4f8ea303993c5e6476659f87235068d46e7ae6a90a370c22"}
Nov 03 01:42:29.269178 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:29.269150    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68v8q/must-gather-xvpbg" event={"ID":"7a457d0a-4dcf-48d7-8e33-f528443f95f9","Type":"ContainerStarted","Data":"efdc3d0f26b1fae0f5c2433a708892a938ee353247b78a0cabe51badbd6bd41f"}
Nov 03 01:42:30.282072 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:30.282029    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/must-gather-95vgz" event={"ID":"d162a947-29f0-4f90-a650-fd109e6633bc","Type":"ContainerStarted","Data":"c8055cdd8ae8de4aa47ccf1055ef2561ea2e999823be8b30ce282a6e8fd59c7d"}
Nov 03 01:42:30.282626 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:30.282088    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/must-gather-95vgz" event={"ID":"d162a947-29f0-4f90-a650-fd109e6633bc","Type":"ContainerStarted","Data":"b0100d9c50164e90202136e3a82c23d2f3999572c3fd7bd80da2c814b21f8d13"}
Nov 03 01:42:33.465918 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:33.465837    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-xhsst_7ad1f241-5476-4a43-a616-f26eff05c43d/global-pull-secret-syncer/0.log"
Nov 03 01:42:34.240066 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:34.239536    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9td9d_21ef2aeb-b54a-45f5-b58a-af731ebdfe11/konnectivity-agent/0.log"
Nov 03 01:42:34.753473 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:34.753442    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-136-97.ec2.internal_b6ab5ae10a035469fb072566acf5cdc5/haproxy/0.log"
Nov 03 01:42:40.432490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:40.431595    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68v8q/must-gather-xvpbg" event={"ID":"7a457d0a-4dcf-48d7-8e33-f528443f95f9","Type":"ContainerStarted","Data":"039920799ab8fe728a7b1d9b92e529728b433713d13c859cfa89bd02cc4ddbab"}
Nov 03 01:42:40.432490 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:40.431651    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-68v8q/must-gather-xvpbg" event={"ID":"7a457d0a-4dcf-48d7-8e33-f528443f95f9","Type":"ContainerStarted","Data":"9e0172cfd97456ff3c2251ab64c59f0314381735226cac4bc989e8d4aebc1e1b"}
Nov 03 01:42:40.476285 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:40.474548    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n428r/must-gather-95vgz" podStartSLOduration=12.545019779 podStartE2EDuration="13.474527675s" totalImagesPullingTime="929.507896ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:30.32155632 +0000 UTC m=+5787.876908345" watchObservedRunningTime="2026-11-03 01:42:40.474527675 +0000 UTC m=+5798.029879703"
Nov 03 01:42:40.476285 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:40.476123    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-68v8q/must-gather-xvpbg" podStartSLOduration=2.083805787 podStartE2EDuration="13.476111359s" totalImagesPullingTime="11.392305572s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 01:42:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:40.472305136 +0000 UTC m=+5798.027657161" watchObservedRunningTime="2026-11-03 01:42:40.476111359 +0000 UTC m=+5798.031463386"
Nov 03 01:42:42.439991 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:42.439963    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kxq2l_f109704e-d9c1-41e3-84c9-5a0aa3eac6be/limitador/0.log"
Nov 03 01:42:43.296611 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:43.296569    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77bbb667f4-ps8vw_ec8b5850-10d2-4e94-96fc-834119b90561/manager/0.log"
Nov 03 01:42:44.140141 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.140106    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6dcd390-f475-4420-a193-e6f8204fc396/alertmanager/0.log"
Nov 03 01:42:44.172378 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.172317    6000 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"]
Nov 03 01:42:44.176775 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.176749    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.182534 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.182511    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-proc\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.182669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.182548    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-lib-modules\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.182669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.182621    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-podres\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.182669 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.182648    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-sys\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.182921 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.182860    6000 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjpjh\" (UniqueName: \"kubernetes.io/projected/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-kube-api-access-qjpjh\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.194693 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.194663    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"]
Nov 03 01:42:44.283306 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.282801    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6dcd390-f475-4420-a193-e6f8204fc396/config-reloader/0.log"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.283564    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjpjh\" (UniqueName: \"kubernetes.io/projected/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-kube-api-access-qjpjh\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.283629    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-proc\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.283677    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-lib-modules\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.283723    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-podres\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.283760    6000 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-sys\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.283910    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-sys\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.284538    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-proc\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.284619    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-lib-modules\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.291287 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.284671    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-podres\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.304015 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.303978    6000 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjpjh\" (UniqueName: \"kubernetes.io/projected/c5cbb0b6-e005-471b-99f5-78eaa0af5e9d-kube-api-access-qjpjh\") pod \"perf-node-gather-daemonset-hbk9d\" (UID: \"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d\") " pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.413065 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.412986    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6dcd390-f475-4420-a193-e6f8204fc396/kube-rbac-proxy-web/0.log"
Nov 03 01:42:44.495868 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.493490    6000 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:44.543054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.543010    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6dcd390-f475-4420-a193-e6f8204fc396/kube-rbac-proxy/0.log"
Nov 03 01:42:44.672642 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.672570    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6dcd390-f475-4420-a193-e6f8204fc396/kube-rbac-proxy-metric/0.log"
Nov 03 01:42:44.797286 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.797241    6000 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"]
Nov 03 01:42:44.798914 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.798891    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6dcd390-f475-4420-a193-e6f8204fc396/prom-label-proxy/0.log"
Nov 03 01:42:44.800882 ip-10-0-136-97 kubenswrapper[6000]: W1103 01:42:44.800857    6000 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc5cbb0b6_e005_471b_99f5_78eaa0af5e9d.slice/crio-1b1f473e7c0b85660d398f6243065375c84dd4cfecc642a47c403e3d5ad1c579 WatchSource:0}: Error finding container 1b1f473e7c0b85660d398f6243065375c84dd4cfecc642a47c403e3d5ad1c579: Status 404 returned error can't find the container with id 1b1f473e7c0b85660d398f6243065375c84dd4cfecc642a47c403e3d5ad1c579
Nov 03 01:42:44.960164 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:44.960077    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_a6dcd390-f475-4420-a193-e6f8204fc396/init-config-reloader/0.log"
Nov 03 01:42:45.261679 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.261594    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2q7ws_dcd7e2a0-6e98-4417-bbc9-25868b1ca911/kube-state-metrics/0.log"
Nov 03 01:42:45.292344 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.292249    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc8c7d7fb-gcc5b_4146cae1-50a3-4d36-b7af-65bb9c50655d/manager/0.log"
Nov 03 01:42:45.365739 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.365696    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2q7ws_dcd7e2a0-6e98-4417-bbc9-25868b1ca911/kube-rbac-proxy-main/0.log"
Nov 03 01:42:45.470291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.469206    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d" event={"ID":"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d","Type":"ContainerStarted","Data":"703f7a473e83eeb3e4e2184711d29be6d61261eb815d0a83b9140f74471bd397"}
Nov 03 01:42:45.470291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.469303    6000 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d" event={"ID":"c5cbb0b6-e005-471b-99f5-78eaa0af5e9d","Type":"ContainerStarted","Data":"1b1f473e7c0b85660d398f6243065375c84dd4cfecc642a47c403e3d5ad1c579"}
Nov 03 01:42:45.470291 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.469351    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:45.493273 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.493234    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-2q7ws_dcd7e2a0-6e98-4417-bbc9-25868b1ca911/kube-rbac-proxy-self/0.log"
Nov 03 01:42:45.506767 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.506713    6000 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d" podStartSLOduration=1.50669978 podStartE2EDuration="1.50669978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 01:42:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 01:42:45.504911605 +0000 UTC m=+5803.060263629" watchObservedRunningTime="2026-11-03 01:42:45.50669978 +0000 UTC m=+5803.062051804"
Nov 03 01:42:45.530289 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.530181    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hm874_6f731881-a91d-48e5-b881-8bee6561c7fb/proxy/0.log"
Nov 03 01:42:45.625592 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.625551    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-c65fccf8-rt7bz_182c2fcb-792e-429f-9ab3-875803a3fd66/metrics-server/0.log"
Nov 03 01:42:45.761075 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:45.761046    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-nv4rq_4c4f0ae1-8981-44e8-ac34-2ee16ee2fae6/monitoring-plugin/0.log"
Nov 03 01:42:46.261946 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:46.261909    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ksp7h_0da10585-07cb-473c-94c2-d9a8c519ea73/node-exporter/0.log"
Nov 03 01:42:46.389294 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:46.389015    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ksp7h_0da10585-07cb-473c-94c2-d9a8c519ea73/kube-rbac-proxy/0.log"
Nov 03 01:42:46.516797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:46.516710    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ksp7h_0da10585-07cb-473c-94c2-d9a8c519ea73/init-textfile/0.log"
Nov 03 01:42:47.006085 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:47.006049    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77bbb667f4-ps8vw_ec8b5850-10d2-4e94-96fc-834119b90561/manager/0.log"
Nov 03 01:42:47.036794 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:47.036763    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-df6xs_a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/kube-rbac-proxy-main/0.log"
Nov 03 01:42:47.165726 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:47.165696    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-df6xs_a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/kube-rbac-proxy-self/0.log"
Nov 03 01:42:47.298827 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:47.298746    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-df6xs_a2de66e0-f1f5-44d4-8eca-3bd623d6fd84/openshift-state-metrics/0.log"
Nov 03 01:42:48.349478 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:48.349392    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-srgkz_5a1b5632-bc8c-4692-8e60-816ef5b35575/prometheus-operator/0.log"
Nov 03 01:42:48.464790 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:48.464759    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-srgkz_5a1b5632-bc8c-4692-8e60-816ef5b35575/kube-rbac-proxy/0.log"
Nov 03 01:42:48.689996 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:48.689964    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc8c7d7fb-gcc5b_4146cae1-50a3-4d36-b7af-65bb9c50655d/manager/0.log"
Nov 03 01:42:48.723877 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:48.723839    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b45cfff7c-f6ktv_50e89fdd-eb0a-44c4-b904-9e7955f88504/telemeter-client/0.log"
Nov 03 01:42:48.852062 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:48.851767    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b45cfff7c-f6ktv_50e89fdd-eb0a-44c4-b904-9e7955f88504/reload/0.log"
Nov 03 01:42:48.867459 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:48.867433    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hm874_6f731881-a91d-48e5-b881-8bee6561c7fb/proxy/0.log"
Nov 03 01:42:48.984085 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:48.984011    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6b45cfff7c-f6ktv_50e89fdd-eb0a-44c4-b904-9e7955f88504/kube-rbac-proxy/0.log"
Nov 03 01:42:49.118269 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:49.118220    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bbb79ccc9-zpjcf_03ed77f9-5e31-428a-988c-7811b60e299a/thanos-query/0.log"
Nov 03 01:42:49.242797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:49.242715    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bbb79ccc9-zpjcf_03ed77f9-5e31-428a-988c-7811b60e299a/kube-rbac-proxy-web/0.log"
Nov 03 01:42:49.370113 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:49.370080    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bbb79ccc9-zpjcf_03ed77f9-5e31-428a-988c-7811b60e299a/kube-rbac-proxy/0.log"
Nov 03 01:42:49.498316 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:49.498208    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bbb79ccc9-zpjcf_03ed77f9-5e31-428a-988c-7811b60e299a/prom-label-proxy/0.log"
Nov 03 01:42:49.631797 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:49.631775    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bbb79ccc9-zpjcf_03ed77f9-5e31-428a-988c-7811b60e299a/kube-rbac-proxy-rules/0.log"
Nov 03 01:42:49.760054 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:49.759982    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-6bbb79ccc9-zpjcf_03ed77f9-5e31-428a-988c-7811b60e299a/kube-rbac-proxy-metrics/0.log"
Nov 03 01:42:49.915124 ip-10-0-136-97 kubenswrapper[6000]: E1103 01:42:49.914912    6000 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-qc4b5" containerName="extractor"
Nov 03 01:42:50.238682 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:50.238651    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-77bbb667f4-ps8vw_ec8b5850-10d2-4e94-96fc-834119b90561/manager/0.log"
Nov 03 01:42:51.489818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:51.489786    6000 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-n428r/perf-node-gather-daemonset-hbk9d"
Nov 03 01:42:51.914481 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:51.914448    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-bc8c7d7fb-gcc5b_4146cae1-50a3-4d36-b7af-65bb9c50655d/manager/0.log"
Nov 03 01:42:52.102916 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:52.102866    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-hm874_6f731881-a91d-48e5-b881-8bee6561c7fb/proxy/0.log"
Nov 03 01:42:53.589105 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:53.589076    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-664c8f59ff-w89r9_416f8d8f-5e16-49aa-b2a5-6140c2bc4ef6/console/0.log"
Nov 03 01:42:54.238950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:54.238906    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-hlqrl_7be7a5b4-d36a-48a6-afc8-40741783c4fa/kube-auth-proxy/0.log"
Nov 03 01:42:54.472178 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:54.472148    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-dhv97_ad946480-1aaa-4a6c-826c-a0db0c4a9ac9/istio-proxy/0.log"
Nov 03 01:42:54.765427 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:54.765395    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-msgxv_f65d6c3e-af31-47a7-a807-15531faca35f/istio-proxy/0.log"
Nov 03 01:42:54.903221 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:54.902839    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-t297d_ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf/istio-proxy/0.log"
Nov 03 01:42:54.970334 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:54.970300    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-w99jj_b869da23-2c4e-4e69-a1b7-a9ae8585d8a1/istio-proxy/0.log"
Nov 03 01:42:55.096667 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:55.096573    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-75b7db44c9-qzv2n_0c0d40bf-fa16-4c0e-aa3c-595693ee1513/payload-pre-processing/0.log"
Nov 03 01:42:55.167063 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:55.167028    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-846db64bd8-6tnsj_9949daa4-550e-4e71-bdb2-7d3f046eeb9d/payload-processing/0.log"
Nov 03 01:42:58.038304 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:58.038269    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv656_8c3c97a4-e610-4a34-9fe8-b5e754372cd2/dns/0.log"
Nov 03 01:42:58.158431 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:58.158396    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rv656_8c3c97a4-e610-4a34-9fe8-b5e754372cd2/kube-rbac-proxy/0.log"
Nov 03 01:42:58.412950 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:58.412846    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6lccb_cb2a078c-bf02-4c8b-aeec-cbd96df8b1a0/dns-node-resolver/0.log"
Nov 03 01:42:58.413187 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:58.413140    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-hlqrl_7be7a5b4-d36a-48a6-afc8-40741783c4fa/kube-auth-proxy/0.log"
Nov 03 01:42:58.617070 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:58.617037    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-dhv97_ad946480-1aaa-4a6c-826c-a0db0c4a9ac9/istio-proxy/0.log"
Nov 03 01:42:58.893342 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:58.893315    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-msgxv_f65d6c3e-af31-47a7-a807-15531faca35f/istio-proxy/0.log"
Nov 03 01:42:59.023346 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:59.023319    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-t297d_ff7bba15-a9a5-4baa-9bc5-5383fcc64bdf/istio-proxy/0.log"
Nov 03 01:42:59.096634 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:59.096599    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-6cc458d887-w99jj_b869da23-2c4e-4e69-a1b7-a9ae8585d8a1/istio-proxy/0.log"
Nov 03 01:42:59.225818 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:59.225506    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-75b7db44c9-qzv2n_0c0d40bf-fa16-4c0e-aa3c-595693ee1513/payload-pre-processing/0.log"
Nov 03 01:42:59.293183 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:42:59.293148    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-846db64bd8-6tnsj_9949daa4-550e-4e71-bdb2-7d3f046eeb9d/payload-processing/0.log"
Nov 03 01:43:00.236146 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:43:00.236086    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wf7kp_fba3da01-3b60-48ec-8d2d-2d90478e334a/node-ca/0.log"
Nov 03 01:43:00.276513 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:43:00.276459    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-kxq2l_f109704e-d9c1-41e3-84c9-5a0aa3eac6be/limitador/0.log"
Nov 03 01:43:03.263613 ip-10-0-136-97 kubenswrapper[6000]: I1103 01:43:03.263573    6000 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-66c6b6454f-hlqrl_7be7a5b4-d36a-48a6-afc8-40741783c4fa/kube-auth-proxy/0.log"